Nuffing.com

by Ajoy Majumdar, Zhen Li

Most large companies have numerous data sources with different data formats and large data volumes. These data stores are accessed and analyzed by many people throughout the enterprise. At Netflix, our data warehouse consists of a large number of data sets stored in Amazon S3 (via Hive), Druid, Elasticsearch, Redshift, Snowflake and MySql. Our platform supports Spark, Presto, Pig, and Hive for consuming, processing and producing data sets. Given the diverse set of data sources, and to make sure our data platform can interoperate across these data sets as one “single” data warehouse, we built Metacat. In this blog, we will discuss our motivations in building Metacat, a metadata service to make data easy to discover, process and manage.

Objectives

The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the metadata service, and the event service. These ideas are not unique to Netflix, but rather a reflection of the architecture that we felt would be necessary to build a system not only for the present, but for the future scale of our data infrastructure.

Many years back, when we started building the platform, we adopted Pig as our ETL language and Hive as our ad-hoc querying language. Since Pig did not natively have a metadata system, it seemed ideal for us to build one that could interoperate between both.

Thus Metacat was born, a system that acts as a federated metadata access layer for all data stores we support. A centralized service that our various compute engines could use to access the different data sets. In general, Metacat serves three main objectives:

• Federated views of metadata systems
• Unified API for metadata about datasets
• Arbitrary business and user metadata storage of datasets

It is worth noting that other companies that have large and distributed data sets also have similar challenges. Apache Atlas, Twitter’s Data Abstraction Layer and Linkedin’s WhereHows (Data Discovery at Linkedin), to name a few, are built to tackle similar problems, but in the context of the respective architectural choices of the companies.

Metacat

Metacat is a federated service providing a unified REST/Thrift interface to access metadata of various data stores. The respective metadata stores are still the source of truth for schema metadata, so Metacat does not materialize it in its storage. It only directly stores the business and user-defined metadata about the datasets. It also publishes all of the information about the datasets to Elasticsearch for full-text search and discovery.

At a higher level, Metacat features can be categorized as follows:

• Data abstraction and interoperability
• Business and user-defined metadata storage
• Data discovery
• Data change auditing and notifications
• Hive metastore optimizations

Data Abstraction and Interoperability

Multiple query engines like Pig, Spark, Presto and Hive are used at Netflix to process and consume data. By introducing a common abstraction layer, datasets can be accessed interchangeably by different engines. For example: A Pig script reading data from Hive will be able to read the table with Hive column types in Pig types. For data movement from one datastore to another, Metacat makes the process easy by helping in creating the new table in the destination data store using the destination table data types. Metacat has a defined list of supported canonical data types and has mappings from these types to each respective data store type. For example, our data movement tool uses the above feature for moving data from Hive to Redshift or Snowflake.

The Metacat thrift service supports the Hive thrift interface for easy integration with Spark and Presto. This enables us to funnel all metadata changes through one system which further enables us to publish notifications about these changes to enable data driven ETL. When new data arrives, Metacat can notify dependent jobs to start.

Business and User-defined Metadata

Metacat stores additional business and user-defined metadata about datasets in its storage. We currently use business metadata to store connection information (for RDS data sources for example), configuration information, metrics (Hive/S3 partitions and tables), and tables TTL (time-to-live) among other use cases. User-defined metadata, as the name suggests, is a free form metadata that can be set by the users for their own usage.

Business metadata can also be broadly categorized into logical and physical metadata. Business metadata about a logical construct such as a table is considered as logical metadata. We use metadata for data categorization and for standardizing our ETL processing. Table owners can provide audit information about a table in the business metadata. They can also provide column default values and validation rules to be used for writes into the table.

Metadata about the actual data stored in the table or partition is considered as physical metadata. Our ETL processing stores metrics about the data at job completion, which is later used for validation. The same metrics can be used for analyzing the cost + space of the data. Given two tables can point to the same location (like in Hive), it is important to have the distinction of logical vs physical metadata because two tables can have the same physical metadata but have different logical metadata.

Data Discovery

As consumers of the data, we should be able to easily browse through and discover the various data sets. Metacat publishes schema metadata and business/user-defined metadata to Elasticsearch that helps in full-text search for information in the data warehouse. This also enables auto-suggest and auto-complete of SQL in our Big Data Portal SQL editor. Organizing datasets as catalogs helps the consumer browse through the information. Tags are used to categorize data based on organizations and subject areas. We also use tags to identify tables for data lifecycle management.

Data Change Notification and Auditing

Metacat, being a central gateway to the data stores, captures any metadata changes and data updates. We have also built a push notification system around table and partition changes. Currently, we are using this mechanism to publish events to our own data pipeline (Keystone) for analytics to better understand our data usage and trending. We also publish to Amazon SNS. We are evolving our data platform architecture to be an event-driven architecture. Publishing events to SNS allows other systems in our data platform to “react” to these metadata or data changes accordingly. For example, when a table is dropped, our S3 warehouse janitor services can subscribe to this event and clean up the data on S3 appropriately.

Hive Metastore Optimizations

The Hive metastore, backed by an RDS, does not perform well under high load. We have noticed a lot of issues around writing and reading of partitions using the metatore APIs. Given this, we no longer use these APIs. We have made improvements in our Hive connector that talks directly to the backed RDS for reading and writing partitions. Before, Hive metastore calls to add a few thousand partitions usually timed out, but with our implementation, this is no longer a problem.

Next Steps

We have come a long way on building Metacat, but we are far from done. Here are some additional features that we still need to work on to enhance our data warehouse experience.

• Schema and metadata versioning to provide the history of a table. For example, it is useful to track the metadata changes for a specific column or be able to view table size trends over time. Being able to ask what the metadata looked like at a point in the past is important for auditing, debugging, and also useful for reprocessing and roll-back use cases.
• Provide contextual information about tables for data lineage. For example, metadata like table access frequency can be aggregated in Metacat and published to a data lineage service for use in ranking the criticality of tables.
• Add support for data stores like Elasticsearch and Kafka.
• Pluggable metadata validation. Since business and user-defined metadata is free form, to maintain integrity of the metadata, we need validations in place. Metacat should have a pluggable architecture to incorporate validation strategies that can be executed before storing the metadata.

As we continue to develop features to support our use cases going forward, we’re always open to feedback and contributions from the community. You can reach out to us via Github or message us on our Google Group. We hope to share more of what our teams are working on later this year!

And if you’re interested in working on big data challenges like this, we are always looking for great additions to our team. You can see all of our open data platform roles here.

---

Metacat: Making Big Data Discoverable and Meaningful at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

via Planet MySQL
Metacat: Making Big Data Discoverable and Meaningful at Netflix

PatSnap, a Euro-Asian company that offers a patent and R&D platform and services, has pulled in a $38 million Series D funding round led by existing investors Sequoia and Shunwei Capital, the investment firm founded by Xiaomi co-founder and CEO Lei Jun. Southeast Asia’s Qualgro also took part.

All three backed the company in 2016 when it led an undisclosed Series C round. While PatSnap didn’t give a figure for that previous round, it is saying this time around that it has raised over $100 million to date. Doing some quick via math via figures on Crunchbase suggests that the Series C was something in the region of $50 million.

PatSnap was founded in 2007 and it is based out of the UK and Singapore, with locations in China and the U.S.. The company started out as essentially a directory for IP, helping companies — and particularly enterprises — pull in data for R&D and product development purposes.

The company claims 8,000 clients worldwide, with the U.S. its largest market for revenue. PatSnap said that in China, its second-largest market and a major focus for the firm, it said it has more than 4,500 clients. In addition to its core service, it is focused on going beyond a data repository to offer services for enterprises that help manage internal product development and other R&D initiatives.

“Patent data let us kick down the door and earn respect, but now we’re looking at completely different products,” Ray Chohan, SVP of corporate strategy at PatSnap told TechCrunch in an interview. “We are working on new products for R&D with a long-term view of becoming the software stack for R&D teams.”

That’s exactly how this new capital will be put to work, Tiong said. Related to that, the company plans to open an office in Toronto, Canada, for development. Already, the company has 700 staff across a range of offices that include London (commercial), China (product), Singapore (machine learning) and LA (go to market).

Series D is a fairly advanced stage for a startup in Southeast Asia (and London) and exits are something that the tech industry is giving more thought to given the growth of the ecosystem, and events such as Sea’s U.S. listing last year. Despite that, Chohan — who founded the company’s London-based office — said that he’s not thinking too hard about the future for now.

“Our obsession is our employees, customers and building great products, if we can do that then the byproduct of a liquidity event will happen by itself,” he explained.

Chohan added that PatSnap is “well funded” and on course to become profitable over the next two to three years.

via TechCrunch
IP platform PatSnap picks up $38M from Sequoia and Xiaomi founder’s fund

Steel Targets can be an awesome addition to any training regime, whether it be for self-defense, long distance shooting, any sort of competition, or just plain bragging rights at the range with buddies. Due to their reactive “CLANG”, steel targets give shooters instant feedback on whether or not the fundamentals are being properly applied. But […]

Read More …

The post Getting the most out of Steel Targets: What You Need to Know appeared first on The Firearm Blog.

via The Firearm Blog
Getting the most out of Steel Targets: What You Need to Know

In 2014, it seemed like pretty much anyone with a pulse and pitch deck was capable of raising huge amounts of capital from prestigious venture capital firms at sky-high valuations. Here we are four years later and times have changed. VCs inked a little more than 3,100 deals in the last quarter of 2017, according to Crunchbase — about 500 fewer than the previous quarter.

For aspiring startup founders, it’s a “confusing time in the so-called Unicorn story,” as Erin Griffith put it in a column last May — an asset bubble that never really popped, but which at the very least is deflating. In the confirmation hearing for new SEC Chairman Jay Clayton, lawmakers lamented the dearth of initial public offerings as companies that thrived in private markets — from Snap to Blue Apron — have struggled to deliver meaningful returns to investors.

This all creates a number of dilemmas for founders looking to raise capital and scale businesses in 2018. VCs remain an integral part of the innovation ecosystem. But what happens when the changing dynamics of financial markets collide with VCs’ expectations regarding growth? VCs may not always be aligned with founders and companies in this new environment. A recent study commissioned by Eric Paley at Founder Collective found that by pressuring companies to scale prematurely, venture capitalists are indirectly responsible for more startup deaths than founder infighting, technical debt and slow customer adoption — combined.

The new landscape requires that founders in particular be judicious in the way they seek out new sources of capital, structure cap tables and ownership and the types of concessions made to their new backers in exchange for that much-needed cash. Here are three ways founders can ensure they’re looking out for what’s best for their companies — and themselves — in the long run.

Take time to backchannel

Venture capitalists are arguably in the business of due diligence. Before they sign the dotted line, they can be expected to call your competitors, your customers, your former employers, your business school classmates — they will ask everyone and their mother about you.

A first-time founder is also new to the pressures of entrepreneurship, of having employees rely on you for their livelihoods. Whether you are desperate for cash because you need to make payroll, or you’re anxious for the validation of a headline-worthy investment, few founders take the time to properly backchannel their investors. Until you can say you’ve done due diligence of your own, your opinion of your VCs is going to be based on the size of their fund, the deals they’ve done or the press they’ve gotten. In short, it will likely be based on what they’ve done right.

On the other hand, you likely don’t know anything about the actual partner that will join your board. Are they intelligent in your space? Do they have a meaningful network? Or do they just know a few headhunters? Are they value creators? What is their political standing in their firm? Before you sign a term sheet, you need to take the time to contextualize the profile of the person who is taking a board seat. It gives you foresight on the actions your investment partner will likely take down the road.

Think beyond your first raise

If you do decide to raise capital, make sure you are in alignment with your board regarding your business plan, the pursuit of profit at the expense of revenue growth, or vice versa, and how it will steer your decision making as the market changes. It goes without saying that differences of opinion regarding your business strategy can lead to big conflict down the road.

As you think about these trade-offs, remember that as an entrepreneur, your obligation is to the existing shareholders: the employees and you. As the pack of potential unicorns has thinned, VCs in particular have turned to unconventional deal structures, like the use of common and preferred shares. For the founder who needs to raise cash, a dual ownership structure seems like a fair compromise to make, but remember that it may be at the expense of your employees’ option pool. The interests of preferred and common shareholders are not perfectly aligned, particularly when it comes time to make difficult decisions in the future.

Is VC money right for you?

VCs frequently share information, board decks and investor presentations with members of the press and the tech community, sometimes in support of their own personal agendas or to get perspective on whether to invest or not. That’s why it’s particularly important to backchannel, and more importantly, that you have allies that you can call on and people who can ensure some measure of goodwill. A good company board cannot be made up of just the investors and you: You need advocates that are balanced and on your side.

These prescriptions can sound paranoid, particularly to the founder whose business is growing nicely. But anything can cause a sea change and put you at odds with the people funding your company — who now own a piece of the company that you’re trying to build. When disagreements arise, it can get tense. They might say that you are a first-time founder, and therefore a novice. They will make your weaknesses known and say you’ll never be able to raise again if you ignore their invaluable advice. It’s important that you don’t fall into the fear trap. If you create a product or service that solves an undeniable problem, the money will come — and you will get funded again.

The term founder-friendly VC was always perhaps a bit of a misnomer. The people building the business and the people planning on cashing in on your efforts are imperfect allies. As a founder and business owner, your primary responsibilities are to your clients, to the company you’re building and, most importantly, to the employees who are helping you do it. As founders we like to think that we have all the answers, especially in bad times. Making sure you have alignment with your investors in challenging and unpredictable situations is critical. It’s important to anticipate how your investors will problem-solve before you give up control.

Venture capital is far from the only way to finance an early-stage business. Founders looking to jump-start their business have a number of alternatives, from debt financing and bootstrapping to crowdfunding, angel investors and ICOs. There are indeed still many advantages to having experienced investors on your side, not simply the cash but also the access to hiring and industry knowledge. But the relationship can only benefit both parties when founders go in eyes wide open.

via TechCrunch
Beware ‘founder-friendly’ VCs — 3 steps founders should take to protect their companies

via Slashdot
NASA Makes Two Decades of Satellite Images of Earth Available To the Public

via Buckeye Firearms Association
Firearm Instructors: List your Ohio CCW and gun classes at the NEW OhioGunClass.org

via MakeUseOf.com
No More Photoshop: 5 No-Signup Image Editors on the Web

via Dilbert Daily Strip
Comic for June 09, 2018

Link

Director Damien Chazelle (Whiplash, La La Land) reteams with Ryan Gosling for a dramatic retelling of the story of Neil Armstrong, and NASA’s early efforts to put a man on the Moon. We haven’t been this excited for an astronaut movie since The Right Stuff.

via The Awesomer
First Man (Trailer)

If you’ve ever seen a picture of a World War II Jeep with brown paint between the points of its “invasion star,” you might have assumed it was just an aesthetic touch. But it was much more than that: the paint was there to keep soldiers safe. Here’s how.

The paint’s purpose was to protect against chemical weapons attacks. It’s called “M5 liquid vesicant detector paint” (a vesicant is a chemical agent that causes blistering.)

After speaking with avid World War II re-enactor, retired U.S. Marine Corps tank mechanic and connoisseur of WWII Jeep canvases and also vesicant paint (he currently sells replicas of the stuff) Farrell Fox—I learned that this paint was actually standardized by the European Theater of Operations in 1943.

A “restricted” government document, which Fox told me he found in the National Archives, describes how the paint works, and how it should be applied to the five-point star (the star was the “National Symbol of all motor vehicles assigned to tactical units” per the War Department’s 1942 document AR 805-5).

“In order to provide for the detection of liquid vesicants, the following procedure is adopted,” the European Theater of Operations document reads. “When the national symbol is located in a position visible to the driver, the space between the points of the star will be painted with paint, liquid vesicant detector, M5, to form a circle around the star.”

“In the presence of a vesicant spray attack,” the document continues, “the detector paint will give warning by becoming discolored from its normal brown to red,” going on to say that the paint does not work if the vesicant is a vapor.

The document continues, saying that if the star symbol isn’t visible, a two-square foot blotch of paint should be painted on an area that the driver can see, and that the paint has to be renewed twice a year or “whenever it becomes checked and chalky.” The paint should be cleaned with a brush and water, and it should be kept away from leaded fuel, the document states.

It’s no surprise that this paint was meant for the European theater, as Germany had become well-known for its chemical weapons use during World War I.

In fact, according to a paper written by the U.S. Army Soldier and Biological Chemical Command, American researchers during World War I developed a paint out of linseed oil paint and a “du Pont lacquer/linseed oil enamel paint,” copying the Germans, who were painting their mustard shells to detect leaks.

(At the same time, it’s unlikely that the paint actually helped save any lives during the Second World War, as the Nazis are known to have used chemical weapons primarily in concentration camps, and not battlefields. Still, given the extensive use of gas during the previous war by the Germans, it makes sense that it would have been seen as a necessary and practical precaution.)

According to the document, the prototype paint turned from yellow to red within four seconds of contacting a mustard agent, but the research was never finished.

The report goes on, saying that after looking at blue British detector paint in the early 1940s, the U.S. army came up with olive drab M5 Liquid Vesicant Detector Paint (it’s worth noting that a number of sources say the paint was more of a yellowish-brown color), with 7.8 million four-ounce cans making their way to the U.S. military by the end of World War II (here’s one original can for sale on Ebay).

The paint was brushed onto various surfaces, and dried within five hours. After that, for the next month, the paint would turn red when met with a liquid mustard agent. (Note that this conflicts with the European Theater document, which says the detector paint generally works for six months).

Some sources online state that the paint is effective at detecting not just mustard gas (which, it’s worth noting, isn’t actually dispersed as a gas but rather as liquid droplets), but also Lewisite vesicant and even nerve agents.

Using paint to detect such chemicals is crucial, as—according to the Organization for the Prohibition of Chemical Weapons—agents like mustard gas, which have a characteristic odor, tend to dull a victim’s sense of smell “after only a few breaths so that the smell can no longer be distinguished.” In addition, the implementing body of the Chemical Weapons Convention states, respiratory damage can occur in the presence of even tiny, unsmellable concentrations of the agent.

So the paint is there to provide soldiers with a detection method other than smell, which is important, because the effects of mustard gas are devastating, with Lifehacker describing them in an article, writing:

Once exposed, victims smell an odor similar to mustard plants, garlic, or horseradish. Soon, they begin to feel intense itching and skin irritation over the next 24 hours. Gradually, those irritated areas become a chemical burn and victims develop blisters filled with a yellow fluid (here’s the least graphic photo I could find). These burns can range anywhere from first-degree burns to deadly third-degree burns. If one’s eyes are exposed during an attack, blindness is also a possibility.

It’s obviously critical for soldiers to be able to recognize the presence of this nasty stuff, which is why the U.S. military used this M5 vesicant detection paint not just on vehicles (more than just Jeeps; thats a Chevy K43 truck hood in the photo above), but also on helmets and on “gas detection brassards,” which Allied soldiers wore on their arms as they invaded Normandy.

via Gizmodo
How Special Paint On The Hood Of The World War II Jeep Protected Soldiers’ Lives