streaming-platform:-apache-kafka-2.7-gets-new-inter-broker-api

Streaming platform: Apache Kafka 2.7 gets new inter-broker API

The Kafka message broker, which has been further developed by the community and Confluent under the umbrella of the Apache Software Foundation, is officially available in version 2.7. The update of the open source project, which is designed for real-time processing of large amounts of data, contains numerous bug fixes, various improvements and some new functions. The Kafka team has made progress especially in efforts to replace ZooKeeper with a self-managed metadata quorum with a view to Release 3.0 (KIP – 500).

Apache Kafka 2.7 offers a new Inter-Broker API for this purpose. It should create the prerequisites for making changes in the In-sync Replica (ISR) faster and more safely. The previous approach, updates of the controller (broker responsible in the Kafka cluster for the status management of partitions and replicas) and the partition leaders in ZooKeeper (znode “https://www.heise.de/brokers/topics/ / partitions / [partitionId] / state “) saved information about zkVersion was accompanied by time delays. In particular, the forwarding of zkVersion by the leader to the controller takes place up to 60 seconds delayed. To prevent future metadata queries from delivering outdated information, the new AlterIsr – API help. It grants the controller the exclusive privilege to update Partition Leader and ISR.

Bill Bejeck from the Developer Relations Team Confluent gives a compact overview of Apache Kafka 2.7

in the video Another measure that should help to replace ZooKeeper is the introduction of a broker-side SCRAM configuration API. As stated in the Kafka Improvement Proposal KIP – 554 in front seen, SCRAM credentials can in future be managed via the Kafka protocol. Version 2.7 of the associated tool kafka-configs has been adapted to the new protocol API.

In the update of the Message Broker there is also a Core Raft implementation in accordance with KIP- 595. A separate “Raft” module contains the core of the consensus protocol. As long as the integration with the controller has not yet been completed, however, Kafka users have a separate server available to test the performance of the Raft implementation.

More details and a complete one The Confluent blog post announcing the new version and the release notes for Apache Kafka 2.7 provide an overview of all new features.

(map)

algorithms:-the-new-image-of-the-old

Algorithms: The New Image of the Old

Rapid advances in machine learning allow a completely new look at the past: Companies such as DeOldify or neural.love use neural networks to clean photos and historical films of scratches, flickering and wobbling, and up to 54 to extrapolate and recolor images per second and 4K resolution. The results are astounding, but historians and media scholars are skeptical, reports Technology Review in its current issue.

The idea of ​​using machine learning to do this To process old pictures and films and to color them automatically is technically obvious. Coloring is, however, much more difficult for an AI than to distinguish between cat and dog pictures, for example. In order to continuously come closer to reality, the AI ​​needs a calculable measure of the quality of the coloring. The simplest way to compare the colored image pixel by pixel with the original color image works rather poorly than right. The catch with this method: “Neural networks have a tendency to cheat,” says Dana Kelley from the start-up DeOldify. They tend to produce “averaged images” as far as possible in terms of color.

Fake or reality? With his software of the same name, which 2018 was published as an open source project, the start-up had its first technical breakthrough that circumvents these difficulties: DeOldify uses a technology that is now mainly associated with deepfake videos: Generative Adversarial Networks (GANs). In the meantime, Antic has licensed a revised version of its software to the genealogy platform MyHeritage – which aims to attract potential new customers by processing historical family photos.

This article is from issue 1 / 2021 of the Technology Review. The booklet is from 17. 12. 2020 available in stores and directly in the heise shop. Highlights from the magazine:

To not only have photos, Instead, transforming moving images from historical films into new-looking videos is even more necessary than true-to-life coloring: the film must be freed from scratches and flickering, extrapolated to a higher resolution, then colored and finally post-processed. In order to automate all of these steps, the Polish start-up neural.love has set up a software pipeline that relies partly on open source solutions and partly on software developed in-house.

No reconstructions The results look amazingly real – but they are completely Not. Therefore neural.love repeatedly emphasizes that the videos from the computer are not reconstructions of the historical material. “We don’t want to replace the old films,” says Elizbeth Peck from neural.love. “But we’re making it more accessible”.

Lorenz Engell, media scientist at the Bauhaus University Weimar, does not want to know anything about the “rhetoric of the closer, of the more tangible”. There is no evidence whatsoever that processed images are perceived by viewers as more authentic, he says. The photo historian Anton Holzer grants the new computer-generated images of the story “a certain fascination”, but from a scientific point of view, he also views the technical processing with due skepticism. “Of course, that slightly distorts the view of the original document,” he says. “Is it our goal to read this historical material without historical breaks, as if what can be seen in the picture only happened yesterday?”

The new picture (11 Pictures) Children deliver vegetables to be sold in a market in Boston. The picture, restored and colored by DeOldify, looks almost idyllic in color …

(Image: DeOldify) (wst)

ossso-brings-cleanliness-and-order-to-open-workshops

OSSSO brings cleanliness and order to open workshops

Want to saw something quickly, but can’t find the saw and quickly vacuum up the wood chips that have been flying around for weeks while searching – anyone who is regularly in an open workshop or hack space probably knows the problem. But instead of always doing the same work, it should be possible to design the systems behind it better. At least that’s what people thought in the hobby heaven of Stuttgart and developed OSSO. The abbreviation is quite self-explanatory for O pen S ource – S ystem for S tidiness and O guideline for open workshops.

Behind this are six sub-projects: osSHELF, osVAC, osCLEAN, osBOX, osTRAY and osWALL, each of which addresses a known workshop problem and is implemented in different ways. For example, under osVAC, the vacuum cleaner concept of the hobby sky was rethought, which had previously often led to poor air thanks to many different vacuum cleaners with different connections and high bag consumption. Now there is a coupling system with a focus on the D 32 – standard hose and a workshop vacuum cleaner with cyclone separator. The resulting print templates for the numerous adapters that combine the existing vacuum cleaner parts are available for download on the Thingiverse 3D printing platform. Further developments are expressly desired.

Please develop further osCLEAN has also been tried and tested in practice Bringing cleaning and cleaning agents together properly. Plaster walls are now hanging in the workshop, on which the devices required, such as hand brushes and cloth rags, can be stored in a space-saving manner. OsWALL should work according to a similar principle. Since the corona pandemic is currently making it difficult to work together, the project is still in its infancy. There are already a few workshop walls with prepared suspensions, but in Stuttgart there is still room for improvement in the options for conversion and reconstruction. Some tools are difficult to assign to their place and when new devices are moved in, reorganization is often tedious.

More order with OSSSO (4 pictures) With osSHELF, different boxes can also be stowed efficiently.

(Image: OSSSO / Hobbyhimmel) Since the project is open source, everyone in Hobby Heaven is happy about anyone interested in developing the templates or implementing their own ideas and uploading their variants to Thingiverse. Short introductions on YouTube have recently been added to the range to make entry into the workshop as easy as possible. There, equipment such as saws, grinding and drilling machines are presented and it is explained what to look out for when using them. The videos may explicitly be used by other workshops and therefore do not replace the introduction on the respective device.

Competition for Makerspaces Finally, we are looking for other, even smaller, projects in our current Makerspace Challenge. Show us your hacks and little helpers against chaos and disorder in the workshop. We will award the three best entries with a basket full of tools and filaments. In addition to the general awesome factor, the focus is on reproducibility and the price-performance ratio. The deadline for submissions via the new Make Projects platform ends on 31. January 2021. (hch)

briefly-informed:-delivery-bottlenecks,-virus-variant,-parcel-delivery,-threema

Briefly informed: delivery bottlenecks, virus variant, parcel delivery, Threema

Delivery bottlenecks for IT products threaten well into the year 2021 Graphics cards, game consoles, webcams: many hardware components cannot be bought at the moment, the warehouses are empty. Reasons for this are the strong demand, triggered by the boom in home work and video conferences as well as problems with air freight, writes the computer magazine c’t in its current issue. For example, some of the international air freight is transported on board passenger planes, of which far fewer have been flying since spring. An easing of the delivery situation for hardware components is currently not foreseeable. Because of the global vaccine distribution, transport capacities will also be scarce in the first half of the year 2021.

Fortress against Corona: EU seals itself off against virus variant With flight bans and border closings, Europe is arming itself against a highly contagious one discovered in Great Britain Variant of the coronavirus. According to initial analyzes by British scientists, the mutation has an unusually large number of genetic changes, especially in the spike protein, reports the German press agency. Germany has therefore largely stopped flights from Great Britain, landings from the country are prohibited from midnight. This is based on an order issued by the Federal Ministry of Transport on Sunday, with the exception of pure cargo flights.

Our weekday news podcast delivers the most important news of the day compressed into 2 minutes. Anyone who uses voice assistants such as Amazon Alexa or Google Assistant can also hear or see the news there. Simply activate the skill on Alexa or say to the Google Assistant: “Play heise top”.

Parcel delivery in favor of inner cities Online retail is the big winner of the Corona crisis. Business in the cities is suffering. To compensate for this, politicians from the CDU and CSU are now proposing a package levy with which an “inner city fund” is to be financed. They advocate introducing a package fee for online trading, “the amount of which is proportional to the order value”. The coalition partner SPD supports the proposal. The FDP and especially the retail sector itself are not very enthusiastic.

Threema apps are now completely open-source The paid messenger Threema becomes open source and thus reveals the source code. “To celebrate we are lowering the app price by 28. December by 50 Percent “, writes the provider in an announcement. Threema already had documentation of the cryptographic process, but now the complete inspection and approval of the code follows. For the time being, it can only be compiled reproducibly under Android.

(sy)

coralibre:-the-corona-warning-app-without-google-services

CoraLibre: The Corona warning app without Google services

The Corona Warning App (CWA), which was or is being developed by SAP and Deutsche Telekom on behalf of the Federal Government, is known to need Google Play Services on Android in order to function. This means that an estimated several million people in Germany cannot install the CWA because their smartphone does not meet this requirement.

This is But not only about people for whom their privacy is particularly important and who have therefore removed all Google services from Android, but also, for example, all customers of the Chinese manufacturer Huawei who do not use Google’s services due to legal requirements of the US government may. In order to solve this problem, a group of developers around the app developer cosee GmbH from Darmstadt is developing the open source app CoraLibre – a fork of the official CWA that is supposed to run on Android phones without Google.

No contact tracing without Play Services Shortly after the presentation of the Contact Tracing API developed together with Apple, Google promised to take care of it that it will eventually get by without Play Services, but that doesn’t seem to have a high priority. The group does not even respond to corresponding inquiries on this topic. So it seems to be up to the open source developers to ensure that more people can use the Android version of the CWA. And that’s no small task. In an interview with heise online , the cosee developers explained to us exactly how the Corona warning app works without Google.

Google has therefore integrated its implementation of the Contact Tracing API into the Play Services in order to be able to update this component of the operating system outside of the normal operating system updates. As it is well known, a significant proportion of all Android devices receive updates very late or not at all, so this makes sense for now. However, this means for the CoraLibre developers that they have to implement the Google / Apple API from scratch. And that’s not all. Because Play Services are also responsible for background activities for Android apps, they also had to develop their own mechanism to keep their app active in the background. This was a problem with very early contact tracing apps, such as those from TraceTogether from Singapore, which had to run in the foreground in order to work and were therefore considered to be quite ineffective in real-life situations. The cosee developers solve this with a notification in the status bar of Android, which is supposed to keep the background process for exchanging the Bluetooth data alive.

The CoraLibre project consists of two separate parts: An SDK, which represents the implementation of the Contact Tracing API, and the actual application. The application is a fork of the CWA, which modifies the code from the upstream project by SAP and Telekom so that it works with the CoraLibre SDK. The CoraLibre developers had first tried to bring their code directly upstream in the CWA project, but submitted topics were put on the back burner because the CWA developers were under time pressure. Accordingly, they now operate their own fork, whose code they want to change as little as possible. According to their own statements, they communicate corresponding modifications back to the upstream developers as early as possible. The aim of the CoraLibre project is to have an app in the end that works as exactly as possible like the official app of the federal government, only on Android devices without Play Services.

Millions of Germans are left out The app could potentially involve millions more citizens in contact tracing. Since politicians and experts repeatedly emphasize how important it is – especially now with the daily rapidly increasing positive SARS-CoV-2 tests in the population and the associated overload of the health authorities – that as many citizens as possible use mobile contact tracing to participate, that seems to be a worthwhile goal. Above all, segments of the population could be reached that have not been involved in digital contact tracing to a large extent.

So far, CoraLibre is still not ready for everyday use and is only available as a test version that has to be installed manually and should only be used on test devices. However, the developers hope to be able to publish a version of the app in Google’s Play Store in the next few weeks. The project is partially funded by public funds, including from the Hessian state government. However, the cosee employees are also very happy to receive support from other open source developers. Interested parties who have experience with Android development can participate in the development of the app on GitHub.

(fab)

threema-is-now-completely-open-source

Threema is now completely open source

As announced in late summer, the developers of the crypto messenger Threema have published the code of the Threema apps as open source. In addition to detailed documentation of the cryptographic process (white paper PDF) and regular security audits, the open sourcing service is intended to guarantee full transparency. As a result, it is no longer necessary to believe the developers’ statements or to trust the assessment of third parties.

The code is subject to the AGPLv3 license, and thanks to so-called reproducible builds, however, can be for the time being only under Android, verify that this corresponds to what the user has installed via Google Play Store or on the device. Thus, for example, manipulated apps can be recognized.

The Threema developers provide the source code via GitHub. The repositories for Threema for Android and Threema for iOS can be found here.

Threema currently costs 1, 49 Euro in the Google Play Store and 1, 99 Euros in the iOS App Store.