Month: June 2018

via MakeUseOf.com
What Is Target Disk Mode? How and When to Use It on Your Mac

Link

Throughout his Ridiculously Self-Indulgent, Ill-Advised Vanity Tour, musician “Weird Al” Yankovic and his band performed a different cover song during each night’s encore, from Manfred Mann’s Do Wah Diddy Diddy to Alice Cooper’s School’s Out.

via The Awesomer
Weird Al Plays 77 Cover Songs

by Ajoy Majumdar, Zhen Li

Most large companies have numerous data sources with different data formats and large data volumes. These data stores are accessed and analyzed by many people throughout the enterprise. At Netflix, our data warehouse consists of a large number of data sets stored in Amazon S3 (via Hive), Druid, Elasticsearch, Redshift, Snowflake and MySql. Our platform supports Spark, Presto, Pig, and Hive for consuming, processing and producing data sets. Given the diverse set of data sources, and to make sure our data platform can interoperate across these data sets as one “single” data warehouse, we built Metacat. In this blog, we will discuss our motivations in building Metacat, a metadata service to make data easy to discover, process and manage.

Objectives

The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the metadata service, and the event service. These ideas are not unique to Netflix, but rather a reflection of the architecture that we felt would be necessary to build a system not only for the present, but for the future scale of our data infrastructure.

Many years back, when we started building the platform, we adopted Pig as our ETL language and Hive as our ad-hoc querying language. Since Pig did not natively have a metadata system, it seemed ideal for us to build one that could interoperate between both.

Thus Metacat was born, a system that acts as a federated metadata access layer for all data stores we support. A centralized service that our various compute engines could use to access the different data sets. In general, Metacat serves three main objectives:

• Federated views of metadata systems
• Unified API for metadata about datasets
• Arbitrary business and user metadata storage of datasets

It is worth noting that other companies that have large and distributed data sets also have similar challenges. Apache Atlas, Twitter’s Data Abstraction Layer and Linkedin’s WhereHows (Data Discovery at Linkedin), to name a few, are built to tackle similar problems, but in the context of the respective architectural choices of the companies.

Metacat

Metacat is a federated service providing a unified REST/Thrift interface to access metadata of various data stores. The respective metadata stores are still the source of truth for schema metadata, so Metacat does not materialize it in its storage. It only directly stores the business and user-defined metadata about the datasets. It also publishes all of the information about the datasets to Elasticsearch for full-text search and discovery.

At a higher level, Metacat features can be categorized as follows:

• Data abstraction and interoperability
• Business and user-defined metadata storage
• Data discovery
• Data change auditing and notifications
• Hive metastore optimizations

Data Abstraction and Interoperability

Multiple query engines like Pig, Spark, Presto and Hive are used at Netflix to process and consume data. By introducing a common abstraction layer, datasets can be accessed interchangeably by different engines. For example: A Pig script reading data from Hive will be able to read the table with Hive column types in Pig types. For data movement from one datastore to another, Metacat makes the process easy by helping in creating the new table in the destination data store using the destination table data types. Metacat has a defined list of supported canonical data types and has mappings from these types to each respective data store type. For example, our data movement tool uses the above feature for moving data from Hive to Redshift or Snowflake.

The Metacat thrift service supports the Hive thrift interface for easy integration with Spark and Presto. This enables us to funnel all metadata changes through one system which further enables us to publish notifications about these changes to enable data driven ETL. When new data arrives, Metacat can notify dependent jobs to start.

Business and User-defined Metadata

Metacat stores additional business and user-defined metadata about datasets in its storage. We currently use business metadata to store connection information (for RDS data sources for example), configuration information, metrics (Hive/S3 partitions and tables), and tables TTL (time-to-live) among other use cases. User-defined metadata, as the name suggests, is a free form metadata that can be set by the users for their own usage.

Business metadata can also be broadly categorized into logical and physical metadata. Business metadata about a logical construct such as a table is considered as logical metadata. We use metadata for data categorization and for standardizing our ETL processing. Table owners can provide audit information about a table in the business metadata. They can also provide column default values and validation rules to be used for writes into the table.

Metadata about the actual data stored in the table or partition is considered as physical metadata. Our ETL processing stores metrics about the data at job completion, which is later used for validation. The same metrics can be used for analyzing the cost + space of the data. Given two tables can point to the same location (like in Hive), it is important to have the distinction of logical vs physical metadata because two tables can have the same physical metadata but have different logical metadata.

Data Discovery

As consumers of the data, we should be able to easily browse through and discover the various data sets. Metacat publishes schema metadata and business/user-defined metadata to Elasticsearch that helps in full-text search for information in the data warehouse. This also enables auto-suggest and auto-complete of SQL in our Big Data Portal SQL editor. Organizing datasets as catalogs helps the consumer browse through the information. Tags are used to categorize data based on organizations and subject areas. We also use tags to identify tables for data lifecycle management.

Data Change Notification and Auditing

Metacat, being a central gateway to the data stores, captures any metadata changes and data updates. We have also built a push notification system around table and partition changes. Currently, we are using this mechanism to publish events to our own data pipeline (Keystone) for analytics to better understand our data usage and trending. We also publish to Amazon SNS. We are evolving our data platform architecture to be an event-driven architecture. Publishing events to SNS allows other systems in our data platform to “react” to these metadata or data changes accordingly. For example, when a table is dropped, our S3 warehouse janitor services can subscribe to this event and clean up the data on S3 appropriately.

Hive Metastore Optimizations

The Hive metastore, backed by an RDS, does not perform well under high load. We have noticed a lot of issues around writing and reading of partitions using the metatore APIs. Given this, we no longer use these APIs. We have made improvements in our Hive connector that talks directly to the backed RDS for reading and writing partitions. Before, Hive metastore calls to add a few thousand partitions usually timed out, but with our implementation, this is no longer a problem.

Next Steps

We have come a long way on building Metacat, but we are far from done. Here are some additional features that we still need to work on to enhance our data warehouse experience.

• Schema and metadata versioning to provide the history of a table. For example, it is useful to track the metadata changes for a specific column or be able to view table size trends over time. Being able to ask what the metadata looked like at a point in the past is important for auditing, debugging, and also useful for reprocessing and roll-back use cases.
• Provide contextual information about tables for data lineage. For example, metadata like table access frequency can be aggregated in Metacat and published to a data lineage service for use in ranking the criticality of tables.
• Add support for data stores like Elasticsearch and Kafka.
• Pluggable metadata validation. Since business and user-defined metadata is free form, to maintain integrity of the metadata, we need validations in place. Metacat should have a pluggable architecture to incorporate validation strategies that can be executed before storing the metadata.

As we continue to develop features to support our use cases going forward, we’re always open to feedback and contributions from the community. You can reach out to us via Github or message us on our Google Group. We hope to share more of what our teams are working on later this year!

And if you’re interested in working on big data challenges like this, we are always looking for great additions to our team. You can see all of our open data platform roles here.

Metacat: Making Big Data Discoverable and Meaningful at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

via Planet MySQL
Metacat: Making Big Data Discoverable and Meaningful at Netflix

PatSnap, a Euro-Asian company that offers a patent and R&D platform and services, has pulled in a $38 million Series D funding round led by existing investors Sequoia and Shunwei Capital, the investment firm founded by Xiaomi co-founder and CEO Lei Jun. Southeast Asia’s Qualgro also took part.

All three backed the company in 2016 when it led an undisclosed Series C round. While PatSnap didn’t give a figure for that previous round, it is saying this time around that it has raised over $100 million to date. Doing some quick via math via figures on Crunchbase suggests that the Series C was something in the region of $50 million.

PatSnap was founded in 2007 and it is based out of the UK and Singapore, with locations in China and the U.S.. The company started out as essentially a directory for IP, helping companies — and particularly enterprises — pull in data for R&D and product development purposes.

The company claims 8,000 clients worldwide, with the U.S. its largest market for revenue. PatSnap said that in China, its second-largest market and a major focus for the firm, it said it has more than 4,500 clients. In addition to its core service, it is focused on going beyond a data repository to offer services for enterprises that help manage internal product development and other R&D initiatives.

“Patent data let us kick down the door and earn respect, but now we’re looking at completely different products,” Ray Chohan, SVP of corporate strategy at PatSnap told TechCrunch in an interview. “We are working on new products for R&D with a long-term view of becoming the software stack for R&D teams.”

That’s exactly how this new capital will be put to work, Tiong said. Related to that, the company plans to open an office in Toronto, Canada, for development. Already, the company has 700 staff across a range of offices that include London (commercial), China (product), Singapore (machine learning) and LA (go to market).

Series D is a fairly advanced stage for a startup in Southeast Asia (and London) and exits are something that the tech industry is giving more thought to given the growth of the ecosystem, and events such as Sea’s U.S. listing last year. Despite that, Chohan — who founded the company’s London-based office — said that he’s not thinking too hard about the future for now.

“Our obsession is our employees, customers and building great products, if we can do that then the byproduct of a liquidity event will happen by itself,” he explained.

Chohan added that PatSnap is “well funded” and on course to become profitable over the next two to three years.

via TechCrunch
IP platform PatSnap picks up $38M from Sequoia and Xiaomi founder’s fund

Steel Targets can be an awesome addition to any training regime, whether it be for self-defense, long distance shooting, any sort of competition, or just plain bragging rights at the range with buddies. Due to their reactive “CLANG”, steel targets give shooters instant feedback on whether or not the fundamentals are being properly applied. But […]

Read More …

The post Getting the most out of Steel Targets: What You Need to Know appeared first on The Firearm Blog.

via The Firearm Blog
Getting the most out of Steel Targets: What You Need to Know

In 2014, it seemed like pretty much anyone with a pulse and pitch deck was capable of raising huge amounts of capital from prestigious venture capital firms at sky-high valuations. Here we are four years later and times have changed. VCs inked a little more than 3,100 deals in the last quarter of 2017, according to Crunchbase — about 500 fewer than the previous quarter.

For aspiring startup founders, it’s a “confusing time in the so-called Unicorn story,” as Erin Griffith put it in a column last May — an asset bubble that never really popped, but which at the very least is deflating. In the confirmation hearing for new SEC Chairman Jay Clayton, lawmakers lamented the dearth of initial public offerings as companies that thrived in private markets — from Snap to Blue Apron — have struggled to deliver meaningful returns to investors.

This all creates a number of dilemmas for founders looking to raise capital and scale businesses in 2018. VCs remain an integral part of the innovation ecosystem. But what happens when the changing dynamics of financial markets collide with VCs’ expectations regarding growth? VCs may not always be aligned with founders and companies in this new environment. A recent study commissioned by Eric Paley at Founder Collective found that by pressuring companies to scale prematurely, venture capitalists are indirectly responsible for more startup deaths than founder infighting, technical debt and slow customer adoption — combined.

The new landscape requires that founders in particular be judicious in the way they seek out new sources of capital, structure cap tables and ownership and the types of concessions made to their new backers in exchange for that much-needed cash. Here are three ways founders can ensure they’re looking out for what’s best for their companies — and themselves — in the long run.

Take time to backchannel

Venture capitalists are arguably in the business of due diligence. Before they sign the dotted line, they can be expected to call your competitors, your customers, your former employers, your business school classmates — they will ask everyone and their mother about you.

A first-time founder is also new to the pressures of entrepreneurship, of having employees rely on you for their livelihoods. Whether you are desperate for cash because you need to make payroll, or you’re anxious for the validation of a headline-worthy investment, few founders take the time to properly backchannel their investors. Until you can say you’ve done due diligence of your own, your opinion of your VCs is going to be based on the size of their fund, the deals they’ve done or the press they’ve gotten. In short, it will likely be based on what they’ve done right.

On the other hand, you likely don’t know anything about the actual partner that will join your board. Are they intelligent in your space? Do they have a meaningful network? Or do they just know a few headhunters? Are they value creators? What is their political standing in their firm? Before you sign a term sheet, you need to take the time to contextualize the profile of the person who is taking a board seat. It gives you foresight on the actions your investment partner will likely take down the road.

Think beyond your first raise

If you do decide to raise capital, make sure you are in alignment with your board regarding your business plan, the pursuit of profit at the expense of revenue growth, or vice versa, and how it will steer your decision making as the market changes. It goes without saying that differences of opinion regarding your business strategy can lead to big conflict down the road.

As you think about these trade-offs, remember that as an entrepreneur, your obligation is to the existing shareholders: the employees and you. As the pack of potential unicorns has thinned, VCs in particular have turned to unconventional deal structures, like the use of common and preferred shares. For the founder who needs to raise cash, a dual ownership structure seems like a fair compromise to make, but remember that it may be at the expense of your employees’ option pool. The interests of preferred and common shareholders are not perfectly aligned, particularly when it comes time to make difficult decisions in the future.

Is VC money right for you?

VCs frequently share information, board decks and investor presentations with members of the press and the tech community, sometimes in support of their own personal agendas or to get perspective on whether to invest or not. That’s why it’s particularly important to backchannel, and more importantly, that you have allies that you can call on and people who can ensure some measure of goodwill. A good company board cannot be made up of just the investors and you: You need advocates that are balanced and on your side.

These prescriptions can sound paranoid, particularly to the founder whose business is growing nicely. But anything can cause a sea change and put you at odds with the people funding your company — who now own a piece of the company that you’re trying to build. When disagreements arise, it can get tense. They might say that you are a first-time founder, and therefore a novice. They will make your weaknesses known and say you’ll never be able to raise again if you ignore their invaluable advice. It’s important that you don’t fall into the fear trap. If you create a product or service that solves an undeniable problem, the money will come — and you will get funded again.

The term founder-friendly VC was always perhaps a bit of a misnomer. The people building the business and the people planning on cashing in on your efforts are imperfect allies. As a founder and business owner, your primary responsibilities are to your clients, to the company you’re building and, most importantly, to the employees who are helping you do it. As founders we like to think that we have all the answers, especially in bad times. Making sure you have alignment with your investors in challenging and unpredictable situations is critical. It’s important to anticipate how your investors will problem-solve before you give up control.

Venture capital is far from the only way to finance an early-stage business. Founders looking to jump-start their business have a number of alternatives, from debt financing and bootstrapping to crowdfunding, angel investors and ICOs. There are indeed still many advantages to having experienced investors on your side, not simply the cash but also the access to hiring and industry knowledge. But the relationship can only benefit both parties when founders go in eyes wide open.

via TechCrunch
Beware ‘founder-friendly’ VCs — 3 steps founders should take to protect their companies

The longest continuous daily satellite observation record of Earth ever compiled is

now available for all of us to peruse

. Tom Yulsman, writing for Discover Magazine:

Multiple instruments aboard NASA’s Terra and Aqua satellites, launched in 1999 and 2002, respectively, have kept close watch on the virtually the entire planet for nearly 20 years. Now, for the first time, the entire treasure trove of imagery and scientific information is available for exploration in Worldview, an engaging, interactive web-based application. I’ve been using Worldview regularly to find imagery for use here at ImaGeo since I launched the blog in 2013. But until now, there was a significant limitation: The data available went back only to 2012. Now, after more than five years of work, NASA has extended what’s available on Worldview back to the year 2000, when the Terra satellite first became operational. Terra was lofted into polar orbit with a suite of five remote sensors. The most comprehensive is an instrument called the Moderate-resolution Imaging Spectroradiometer, or MODIS.

via Slashdot
NASA Makes Two Decades of Satellite Images of Earth Available To the Public

via Buckeye Firearms Association
Firearm Instructors: List your Ohio CCW and gun classes at the NEW OhioGunClass.org

via MakeUseOf.com
No More Photoshop: 5 No-Signup Image Editors on the Web

Transcript

Wally: In my right hand is a standard drone. In my left hand is a drone using the cloaking technology I invented. Voices: Ooh! Wow! Wally: I’ll demonstrate it flying as soon as I finish the noise cancellation. CEO: Employee of the year!

via Dilbert Daily Strip
Comic for June 09, 2018