Intel at the Spark + AI Summit: Purpose-Built AI for Intelligence Everywhere
We’re in an Artificial Intelligence (AI) revolution. You can’t scan the Internet without finding market predictions, round tables, and a rush on data scientists. AI is a buzz word, much like big data analytics before it. Interestingly, a recent survey of Fortune 1,000 executives found 76.5% saw big data and AI practices as being intertwined.
That makes sense. Developing a solid data strategy is a challenge and starts with collecting, prepping/cleaning, and managing enterprise data, and then running analytics, which may include machine learning elements. And AI development has surged due to the massive amount of data now available through big data efforts.
Intel has been delivering breakthrough advancements and optimizing analytics workloads for decades, including in-memory database and Apache Spark*. Now we are building AI technologies specific to requirements in the data center, cloud, edge, and devices. Today I want to highlight some of this work.
Getting Started: Existing Infrastructure and the Intel® AI Portfolio
Companies with big data practices can use existing infrastructure as the foundation for their AI plans. Thinking about the path from opportunity to real-world use, it’s important to consider the total time to solution.
AI training is a small but important portion of the workflow, and Intel has optimized Intel® Xeon® processors for deep learning (DL) with a combination of generational hardware improvements and software optimizations within generations. Training performance is currently up 127x and inference performance up 198x versus legacy platforms.
Intel also offers a broad portfolio to ensure customers and developers have what they need to solve AI challenges. This includes hardware, such as Intel® Xeon® Scalable processors for cost-effective AI cloud instances and efficient on-premise utilization for both AI and more standard data center workloads, as well as Intel® FPGAs for inference acceleration, the upcoming purpose-built accelerator Intel® NervanaTM Neural Network Processor, and the Intel® MovidiusTM Visual Processing Unit for low-power AI in devices.
Intel software solutions include Intel Libraries, such as the Intel® nGraph™ Compiler that enables frameworks to use any hardware with peak performance. Additionally, optimizations aim to ensure all major deep learning frameworks and topologies run well on Intel® architecture (IA), including TensorFlow*, Caffe*, MXNet*, and Chainer*. Finally, Intel productivity tools include the Intel® Deep Learning Deployment Toolkit and the OpenVINOTM toolkit.
Intel and the Apache* Spark Ecosystem
Apache Spark has emerged as a great choice for data analytics, with Intel Xeon processors powering a large portion of workloads. Spark fills a role at the intersection of AI, streaming analytics, and batch analytics, offering a path for AI on big data and ease of use for developers writing in Java*, Scala*, or Python*. It’s not surprising that Spark has been widely adopted with expanding enterprise use.
Intel is a top contributor to Spark and has worked with the ecosystem to open source a number of packages, such as Streaming SQL, WebScaleML, and BigDL, as well as contribute to and optimize Spark components for IA. We’ve also helped accelerate Spark queries with the Optimized Analytics Package.
Intel at Spark + AI Summit
Intel is excited to be a Diamond sponsor of the upcoming Spark + AI Summit. We will be leading 10+ technical sessions and showcasing innovations in Apache Spark and AI on topics such as building reinforcement learning applications, compression/decompression codecs on Spark, and accelerating DL training with BigDL.
BigDL is a distributed deep learning library for Spark that runs on existing Apache Spark or Apache Hadoop* clusters, bridging advanced analytics and AI practices. At the summit, Intel will highlight use cases on Using Crowdsourced Images to Create Image Recognition Models with BigDL and Using BigDL on Apache Spark to Improve the MLS Real Estate Search Experience at Scale.