SparkSummit

Spark Summit East NYC
#SparkInsight @theCube conversation thought leaders in Spark during Spark Summit.
   4 years ago
#SparkSummitSpark Insight SF#SparkInsight @theCube conversation thought leaders about In-memory & Spark during Spark Summit.
Bert Latamore
@DataBricks CEO @AliGhodsi live now on #theCUBE from #SparkSummit East. Watch, post comments at http://bit.ly/1TmHT3...
[LIVE CHAT] Spark Summit East NYC
#SparkInsight @theCube conversation thought leaders in Spark during Spark Summit.
Bert Latamore
We were all active in #hadoop. Saw that simple ops were very complicated there. @AliGhodsi on #theCUBE
Bert Latamore
Our vision was it won't be this way in 10 yrs. We wanted to be part of that journey. @AliGhodsi on #theCUBE
Bert Latamore
@DataBricks has done a lot of training efforts. @DVellante on #theCUBE
Bert Latamore
Great training material is part of empowering users to use data. @AliGhodsi on #theCUBE
Bert Latamore
We wanted to lower the bar & make self-service data analysis happen seamlessly. @AliGhodsi on #theCUBE
Bert Latamore
One beauty of the Cloud is it is elastic. So you can do projects much easier than on prem. @AliGhodsi on #theCUBE
Bert Latamore
We often get customers who have extensions. Often there's a cloud provider that has solved the problem, so we just incorporate them. @AliGhodsi on #theCUBE
Bert Latamore
You have 1 engine, not 30-odd projects on different release cycles. @GGilbert41 on #theCUBE
Bert Latamore
We only have 2 versions running in the cloud at any time. We release every week. @AliGhodsi on #theCUBE
Bert Latamore
On prem you can't do that. Lots of old versions running, release cycles in months. @AliGhodsi on #theCUBE
Bert Latamore
Cloud allows continuous feature develop & release. @AliGhodsi on #theCUBE
Bert Latamore
Developed a million lines of code now in open source. How do you draw the line on what is open source? @AliGhodsi on #theCUBE
Bert Latamore
Things you run on our service on top of #Spark you should be able to run on any other Spark version. @AliGhodsi on #theCUBE
Bert Latamore
THings that run out of the box don't need to be open source. @AliGhodsi on #theCUBE
Bert Latamore
We developed SQL for Spark and donated it to open source. That was a good decision. @AliGhodsi on #theCUBE
Bert Latamore
What is the business impact of simplifying #Hadoop? @DVellante on #theCUBE
Bert Latamore
What does #Spark bnring to solving the business problems? @DVellante on #theCUBE
Bert Latamore
Customers don't care about tech. They have a problem they need to solve. @AliGhodsi on #theCUBE
Bert Latamore
The vision is giving customers something that works end-to-end. @AliGhodsi on #theCUBE
Bert Latamore
Unifying different use cases. @AliGhodsi on #theCUBE
Bert Latamore
When we developed Spark one problem was customers had data in different places & in different forms. @AliGhodsi on #theCUBE
Bert Latamore
So it was the scale, the variety & different use cases. @AliGhodsi on #theCUBE
Bert Latamore
Customers want forecasting, anomaly detection, cluster @AliGhodsi on #theCUBE
Bert Latamore
And custoemrs want to do it all in real-time. @AliGhodsi on #theCUBE
Bert Latamore
So eliminating data silos, giving customers real-time while supporting several use cases is the goal. @AliGhodsi on #theCUBE
Bert Latamore
Data has become democratized. You have to keep the vendors honest. Companies now can see if they get the value they r paying for. @AliGhodsi on #theCUBE
Bert Latamore
They know more about me than I do. That creates value but is a little scary. @DVellante on #theCUBE
Bert Latamore
Business is figuring out what you need before you know. As long as it is machines, not humans, reading your data. @AliGhodsi on #theCUBE
Bert Latamore
Customer Panel live now on #theCUBE from #SparkSummit East. Watch, post at http://bit.ly/1TmHT3...
Bert Latamore
@WhiteOps focuses on bot detection, verify there's a human on the other end of the streasm. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
$7B lost to advertisers annually. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Constantly developing these algorithms that run in stream & in batch. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
#DataXu provides greast media to ad agencies. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
Have tons of data. Lately switches to Spark. #DataXu’s Beth Logan on #theCUBE
John Furrier
Beth Logan is giving some great examples
John Furrier
@dvellante what's the impact for spark on their businesses
Bert Latamore
@TerbiumLabs provides a service that traces data leaks for companies. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
Industry changes every month. Spark gives us the flexibility to change with the market without rewriting everything. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
Lots of use cases for malware #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Widescale ad fraud is a lot of easy money and a recurring model for cybercrime. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Any time you get that kind of money you have lots of adaptation to get around detection mechanisms. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
We use evidence based analysis collecting 2000 datapoints on the operation of each browser to identify data leakage. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
We see 10-15B events a day on the Web. 20 TB a day. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Our analytics platform is within 3 minutes of real time. Another platform we have is milliseconds #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Defense while necessary is no longer sufficient. Given the number of different threats you have to assume the data will get out. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
We look beyond the enterprise's borders for fingerprints of stolen data. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
If we can bring discovery time down from hundreds of days to minutes you can provide a lot of value. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
#DataXu started wtih Hadoop and moved to Sparks. #WhiteOps started on hadoop and is moving to Databricks. #TerbiumLabs @GGilbert on #theCUBE
Bert Latamore
#TerbiumLabs started on hadoop. Could take several hours or overnight to run a basic query. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Starting to build our database. It looks fast but the jury is out. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
For #DataXu the Spark win is time to production for applications. That is a big win. Probably a shift from months to weeks. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
#TerbiumLabs had a similar experience. Big Python shop. We're shooting for sub 15 min from event to notifying clients. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
We focus more on speed and agility rather than deep analysis. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
We run on AWS S3 and putting Spark on top of that. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
#Spark's value spans the system. It's not just faster analysis. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Being able to write in Python, SQL & dip into R is powerful. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
Our apps have to run real time on a system that does 2.6 M decisions a sec. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
All these technologies are still new. Things aren't always documented completely. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
It's only going to get better as we go forward. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
5 yrs ago a unified platform wasn't even a question. Now it is possible to have a unified data platform. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
My dream is that I won't need 7 or even 3 databases. #WhiteOps CTO Tamer Hassan on #theCUBE
Bert Latamore
My dream is that our customers could bring their own algorithms to our system & dthey would run. #DataXu’s Beth Logan on #theCUBE
Bert Latamore
My dream is limitless scaling. #TerbiumLabs CEO Danny Rogers on #theCUBE
Bert Latamore
We've been building these communities for 5-6 yrs. My hope is that we can continue to leverage them to get info to help doers accomplish what they need. @DVellante on #theCUBE