#254 Allen Day: Google's Mission to Provide Open Datasets for Public Blockchains

Transcript

hey we're core organizing a live event at the San Francisco blockchain week it's called SF blockchain epicenter and will be October 8th and 9 at the Hilton Union Square you come see members of the epicenter team and a lot of familiar faces from the show there are reduced rates for developers and you can learn more at SF blockchain week I owe this episode of epicenter is brought to you by Dutch X the fair and secure decentralized exchange platform bike Knossos to learn how you can build

apps which leveraged excess liquidity cool visit epicenter TV slash x and buy microsoft azure configure and deploy a consortium network in just a few clicks with pre-built configuration and enterprise grade infrastructure spend less time on watching scaffolding and more time building your applications learn more at AKA OMS slash epicenter hi welcome to epicenter show which talks about the technologies projects and start driving decentralization and the global blockchain revolution my name is Emma thank you Jill and today I'm very pleased to have with me Alan day who is science advocate at Google at the singer

for office we met in Singapore a few months ago when I was travelling through Asia and at the time he told me about this really interesting initiative which I hadn't heard about but it's been had been up for a few months which was that Google had actually added the Bitcoin entire Bitcoin transactional data set to their cloud infrastructure and was on the cusp of releasing also an aetherium data set and so you know I vowed to have him on at some point so we could discuss this and in August Google didn't of fact release their aetherium data set on on bigquery so I'm here with Alan today to

talk about but all this and other things so Alan thank you so much for having up for coming up yeah sure it's really my pleasure to be here so before we get started let's talk a bit about your background your your PhD is in human genetics talk a bit about your journey where do you where did you come from and how did you end up at Google working as a science advocate putting blockchains in big datasets yeah I've been working with computers since I was a little kid and when I went into my as I as I was moving through school and eventually ended up in a doctorate program I was I was combining computing and biology all

the way through that and so that led me into a dangerous disciplinary field called bioinformatics and that involves working with distributed systems for doing scientific computing as well as you know large data sets and computer science and statistics and so I I was becoming something that's now called the data scientist before the title really existed a lot of these people come from physical sciences and so you know once I once I'd acquired that skill set it was quite easy to apply it to to other disciplines and so I could see that there was something interesting happening with these blockchain datasets and so I decided to start looking at

those and applying some of the same techniques and methods that I'd learned for analysing biological networks to analyze these new types of financial networks so one one thing that I think is sort of interesting and I think I think may heard my co-host has often talked about this is this idea that blockchains and biological systems are so much similar or they have common characteristics in that a blockchain can get mutate and can fork and you'll have a sort of new evolution within its within its life form you maybe give us your take on what your thoughts are on this do you find that there are

similarities between the way blockchains evolve to the way biology has evolved yeah certainly the the most the most direct parallel is the for king that happens between projects where one project may decide to change the operational rules for how the consensus works for example or the block timer block size or something and that's very similar to if you were to have a mutation that caused two populations of individuals from the same species to become different species so a speciation event is the equivalent of a fork also if you look at the smart contract

platforms and having smart contracts throw it on chain and these things have some function that's made available to any blocks that are added after the smart contract was added there's some additional effects that are possible as the as the blockchain evolves that's also related to you know adding new functions into a genome for example yeah there are certainly some parallels do you know of anyone who's doing any research on this and that is sort of exploring this at a much deeper level no but there was a there's something interesting that I encountered so from a friend of mine he his name's Daniel Suarez he's a science sci-fi author and he recently published a book called

change agent which is about a bioinformatician based in Singapore so I thought that was kind of interesting since system parallels with my life there but in his book he he talks about block chains at some point and one of these chains is called a bio coin which the proof-of-work will just call it that is is basically blocks are added as a result of some mutation happening in a bioreactor and so there's some there's some interesting a concept or idea here that if you could define a fitness function that you wanted to have a population of organisms move toward through directed evolution that's some form of work because you're exploring

the combinatorial space of the genome or the proteome or whatever aspect of the these living systems that are you solving in parallel right you're trying to move toward some target and that's the work that you're establishing if you have a way to measure that you could actually link evolution to to adding records on to onto a chain so this is maybe a way that we could do some interesting work but as part of securing the chain but it it requires a much lower cost of genome sequencing and genome editing than we have today but certainly in the future if you look at the rate at which these things there are dropping in cost it's conceivable some kind of technology like this could could

could exist that's that's really fascinating I think we could probably spend the whole up so it's exploring this topic but specifically I want to have you want to discuss this this initiative as Google has had bringing the Bitcoin in theory and blockchain onto onto Google Cloud but first you know tell us about your role at Google what is it what what's typical day like as a science advocate at Google sure so we can start with by unpacking my title a little bit so this is one that I just gave to myself because my official title is a developer advocate and I'm specifically interacting with communities who are involved in mostly physical sciences and part of that is

doing communication and so this is more a title that resonates with them so I usually just use that my day-to-day is as I mentioned communications so doing interviews like this or blogging or public speaking this is maybe I don't know 30 40 percent of my time about half of my time I spend doing software development so I'm actually an engineer at Google but I happen to be externally facing showing people outside what cloud can be used for to develop interesting applications and then collecting information from outside seeing what the market is doing what kind of cool stuff people are building and in particular where they're encountering friction or where cloud doesn't have some specific

offering it and then bringing that back into Google to help product teams to help us make better stuff for the people that we're trying to serve and the remainder my times you know like everyone else administrative kind of thing quite a lot of travel and email and etc and so this is avocation work that you do is mostly centered around cloud platform or do you also touch other Google products it's it's all cloud yeah and I'm specifically building things that are more like end-to-end realistic use cases and I'm working quite a lot with these public datasets as we'll talk about later I'm sure some of my my colleagues are doing more like feature advocating about you know incremental updates on products but I

tend to build large integrated projects that are touching many possible cloud components hmm interesting so you know as you said you you you build these projects that sort of more realistic and they're sort of like experiments that could potentially turn into products as it has have any of the things that you've worked on turn into sort of morphed into Google products or anything that has been commercialized I give a lot of feedback too so as a geneticist I work quite a lot with the the genomics and health care product team and yeah definitely some of the stuff that that I I encounter I their fiction that I'm encountering as an individual developing

with the tools I'm basically a customer zero I give them that feedback and then stuff that's really bothering customers I give that to them too and then that that results in updates to products yeah for sure it sounds like a really fascinating role where you can you know live your your passion for technology and science while experimenting and having sort of a lot of flexibility to propose new types of absolutely yeah it's like kernel II yeah people who love to play with new technology um they find themselves quite at home in this kind of role and I basically just to play with get to play with like Lego bricks all day it's fantastic yeah that's cool so let's let's talk about Google cloud platform at a high level so I think most of our listeners will

probably be familiar with Google cloud platform so but you know give us a high-level overview of that product and the types of components that exist within it okay sure yeah it's a public cloud so we have a bunch of data centers and network connecting the data centers it's a comprable to other public clouds in that regard Google spent operating its own data centers for 20 20 years now we just passed our 20th birthday and so we know quite about quite a bit about how to operate these things the first cloud product was something called App Engine which is is still around was a bit ahead of its time today our product areas you could break it down roughly into three three areas one of them is related to virtualization

and infrastructure so this would be kubernetes or other types of virtual machine services and micro services infrastructure networking firewalls etc another area product is relate to applications development so this would be for App Engine fits into there for example or other components for building let's just say Web Services and integrating all of your stuff together to make something usable and then the final area is data analytics and this is the the area where I'm advocating which is primarily Big Data technologies so bigquery is one of these a BigTable spanner are a bunch of our databases most a lot of put a lot of its data

database related and in storage and then on storage and then on the compute side we have a whole bunch of AI technologies and so you know you can't really compute if you don't have data and data is not really useful if you to have compute so we we have these two things that can move data back and forth between them to build new data from old data and the more they're more interesting types of services we have in the compute side are our AI related give us a sense of how big this cloud computer is I mean I don't know what kind of metric we want to use if it's whether it's a number of data centers number of computers or you know had a bias of information processed do you give us a sense of how massive Google Cloud is yeah I can't give you

like specific stats on the number of data centers but I know that we have were represented in all the all the major geographies around the world and have our own dedicated connection between the data centers so the connectivity is quite good running through our dark fiber doesn't ever pass over the public Internet and then a lot of our services are because again it comes from this heritage of Google before Google cloud we have built some services like spanner for example so this is a globally consistent distributed database that relies on atomic clocks to make sure transactions are allowed these different data centers to be synchronized and that's now

available via a public cloud as well so there's a whole bunch of like goodies from Google that are that are inside of this public cloud there's a lot of big customers on here a snap chat runs on Google cloud for example if you remember Pokemon go people still play this that's also in Google Cloud Dropbox runs on Google cloud yeah there's a lot of customers illah runs a Google cloud they do they do some interesting a eye stuff related to forecasting and image analysis of properties geospatial analysis yeah it's it's it's quite big and we've got some major customers now you mentioned the Google cloud was sort of a suite of services so we've got the virtualization aspect we've got the

storage aspect and then also the data processing and machine learning in AI as a user of Google cloud I presume that you can sort of beat all these products integrate together correct yeah yeah there's some places where some you know some components don't interact as seamlessly as you would like or to move data between them but in general there's some way for them to interoperate hmm so what are the the most interesting things of most cutting edge things that you've seen that you can talk about the people are doing you know in areas like like data processing your research or things that people are doing with your AI modules oh I would I

would recommend looking at another YouTube channel called two minute papers this is a a they're usually a little bit more than two minutes but they cover the latest advances in deep learning and quite a lot of that is happening with a application sorry an SDK called tensor flow and tensor flow was developed by Google it was open source this is the largest largest most popular library for doing deep learning which is the the current most popular area of machine learning and that's that's all compatible with Google cloud it all runs the Google cloud we've got specific services that make it run really really well cloud machine learning engine for example

yeah there's computer vision is one of the one of the most interesting areas you could see how computers are able to now drive cars for example right so they're doing real-time analysis of images and looking at all the sensor data coming in and using that into the model of how the car is operating to make sure it can operate safely so moving now towards more towards the the bigquery component can you spend a bit of time describing bigquery and the different components there as it relates to what what you're now doing with Bitcoin and aetherium sure so bigquery is also a distributed system similar to spanner as I mentioned earlier but it's not distributed across multiple data

centers like I was describing its living more locally than that but it still has a whole bunch of nodes that store parts of a data set and so when you do a query you're actually running a job in parallel across a large number of machines to produce a result and so we take the approach basically having data center as computer and don't try to implement anything very fancy like indexes on the tables we just do linear scan across everything because we have enough hard drives that it's it's a economical enough to do that given we can distribute the workload well within to the data center and we're using AI to do

that and because we're not making any assumptions about the structure of the data set it's quite workable for many different data sets of many different shapes and sizes and scales extremely well you know the Dutch have given us so much orange carrots Bluetooth artificial hearts even doughnuts were invented by Dutch people but they also gave us Dutch auctions which as it turns out are great for decentralized exchanges Dutch X is a decentralized trading protocol for er C 20 tokens and it's invented designed and built by kenosis current order boat based exchanges whether centralized or decentralized have a couple of issues miners and exchanges can frontrunner trade when they step in front of a large order to

gain an economic advantage not to mention issues with securing funds by listing fees lack of liquidity and pricing efficiencies the Dutch tax exchange platform uses a Dutch auction mechanism to determine the fair value for a token and participants in a trade are encouraged to reveal their true likeness to pay which eliminates front running as a permissionless Unchained protocol it's useful for BOTS and other smart contracts needing through exchange tokens and Dutch checks also acts as an oracle for - requiring a price feed so to learn more check out the documentation at epicenter TV /x smart contracts are live on a theory main net so you can start building today we'd like to thank gnosis and Dutch X for their support of epicenter so people are using bigquery with their own data sets

so presumably all types of companies from you know companies processing consumer data to you know user user behavior data you know whatever types of data that one can think of wanting to process you could presumably use bigquery to hold that data and and query the data to get your some sort of analysis but there are also public data sets on bigquery and this is specifically in the context of what we're talking today quite interesting because there are I think quite a few quite a few public datasets on there can you describe some of the other datasets

that people are using on bigquery yeah there are there's as you mentioned there's this quite a lot of private datasets and then those can be joined against public datasets for we could call it augmentation for example where you might have some private information that you want to enhance or enrich by joining against public data the majority of the public datasets though are not not not dynamic so they're not regularly updated it's typically some kind of toy data set like a there's one about the New York taxi cabs and so there's a I forget how many days or months it is of data but it's a snapshot or a sampling of taxi rides what time the pickup happened what time the drop-off happened

and what was point A to point B there's other ones that look at types of trees and shade coverage for doing solar radiation analysis on on city streets there's various image datasets this one from I forget which museum it's from but a bunch of pieces of art are cataloged another data set I produce was a genomics data set a 1000 different cannabis genomes where in order to accelerate innovation in an agriculture there's a whole bunch of stuff happening right now with this new plant that is undergoing regulatory changes and so if we begin to look at the the genetic structure of these plants we might be able to improve the varieties more

quickly so there's a whole you know hodgepodge of all kinds of different stuff weather data satellite imagery data all of the reddit comments are also in in bigquery so if you wanted to query any of the the subreddit and threaded forums you can look at all that all of github is then using bigquery not just the source code bail for the comments and the merge requests and everything so if you wanted to do some code analysis that's a pretty popular one because you know developers are interested in development so it gets quite a lot of quite a lot of use I like this idea of combining private datasets and public datasets and you know some of the ways that one might use that so for example tell me if this makes sense like

your for your company like like a ride-sharing app and you want to gain some insights through the you know people are using your app and specifically with regards to your competition which are taxis you know you could use that New York taxi ride dataset and cross it with your own dataset of like how your users are using your app how many times a day or week they're booking rides and then maybe extract some sort of insight from that so that you can perhaps put more cars in a certain area to better compete with with the New York taxis for example one of the types of examples can you point

to as to how people are using you know public datasets you know with their own private datasets sure yeah conceivably that's possible although bear in mind that this taxi dataset was we can keep working with this example is it's quite small and limited in what it has the yellow cab company is not floating all of their data into the public data so it's just a little bit as a toy but that raises an interesting possibility what if all of the data were available what would it you know what how much would you have to pay to incentivize the company to put all their public data out there or at what level of resolution which you would you be willing to pay for lower resolution data and would they be willing to sell that as opposed to

the highest resolution data and so there's actually an interesting case study we can provide it as a supplement possibly in the in the comments or something that's thomson reuters did something like this where they actually host their headline data along with some other attributes i don't know if it's the full article or what I've not looked at it it's a private data set and what they're doing is they're using Google data exchange to make this available using Google's access control so Google basically allows them to manage the access control and is managing bigquery tables to store the data such that Thomson Reuters has to only take the responsibility to put the data in and

then they're selling subscription access to get access to these tables so you could do this you could also put data into into into queues for real-time streaming analysis so that's an example of where you know we can now generalize out to not just two data sets but actually having the notion of a marketplace and maybe there's some opportunity for transportation or logistics companies to bring it back to yellow cab where they could be willing to operate by exchanging some or all of their data and pricing it accordingly depending on you know how much if you need access to how much latency you're willing to accept etc so turning all those knobs that it

all is possible in in a marketplace design you could think about ad tech that's doing a very similar thing right like advertising and ad exchanges it's quite a similar idea interesting well maybe we can come back to some other examples of it learn later on in the show so let's let's talk about this this Bitcoin blockchain data set on on big cloud so this was this came out in February of this year so what was the what what what exactly does it includes what is the Bitcoin data set on on Google's cloud on on bigquery and what was the goal in making this data set publicly available I wanted to be able to explore the data and sort of in my own selfish interest

to be able to make some blog posts or just look at the data because I know other developers were wanting to do this too you certainly see a lot of interest in in cryptocurrency and Bitcoin and any of these keywords they're all growing over time right it's like okay there's developers here I can go and de comer these developers and draw some attention to Google Cloud and I know we have good AI tools so blockchain plus AI should be like super exciting right so I tried to do some of these queries and it turned it out to be really difficult to talk to a Bitcoin node directly and usually the kind of query I'd want to do with some kind of historical analysis and that's not possible just going block by block very easily you have to query one by one by one whereas normally in a sequel type

of scenario you do like a group by two to aggregate it's a particular type of operation for this kind of programming and so I realized that I could extract these data out of the Bitcoin data set and put them into bigquery and and do these analysis I wanted to do and so that's what's in there it's nothing more than the coin blockchain data itself so we download all of the blocks it's a about 200 gigabytes worth of data and then parse each of the blocks and load it into into bigquery so every time a new block comes out we update the table and put it in there it's just the transaction data I don't know how much your listeners would be familiar with what's in Bitcoin but it's really just

some addresses are sending some number of Satoshi's from address a to address B right so I'm looking at us right now so I will link to the shona in the show notes to the the bigquery Bitcoin data set so it actually has two tables so it has a blocks table and a transactions table correct yep that's right and the transaction those actually identical tables the reason I did that is ID normalized it because the way the bigquery pricing model works is that you're paying for unit of i/o and if I uh nest the blocks I allow access to the transactions at a lower cost so it saves the users money to do it that way okay that makes sense right so rather

than querying simply the the rather than having to query the block and then find the transactions within the block then you can simply query transactions yes correct that's right okay that lis interesting and so you update this every every time a block is confirmed every block we're staying intentionally six blocks behind height because that allows us to avoid having to deal with chain rewards we don't want to have to delete data there's some complexity right because if you add a block and that ends up not being the real chain and just like some kind of dead branch on the chain you don't want to have to then delete that and manage manage that you'd rather if you're going to build the simplest

possible system you don't want to take that into consideration and so by staying several blocks behind the tip of the chain you can avoid that problem but the trade-off is then the data are slightly stale okay so you're not storing orphaned orphan chains or orphaned block correct yeah none of the branches on the block chain that don't end up becoming part of the main trunk are not stored in the table okay interesting is there a particular reason why you chose not to also store that data it seems like there could potentially be some interesting analysis that could be made on on this on orphaned blocks well any any data that's on an orphan block is not part of the

consensus right there was some minority thought there might be a block there that everyone had agree to but due to race conditions or randomness or whatever it just ended up not not being the case looking at what transactions end up on these dead branches yeah it could be interesting maybe maybe there's some censorship happening on the blockchain where somebody's blocking you know entity a is blocking entities B's transactions from being placed on chain by denial of service attack in them I suppose so yeah it could be interesting I have not I have not gone down that direction that's real that's actually a really cool idea though yeah I was thinking swerve along those lines right I mean if if at some point that were to

become be the case perhaps we could detect those types of an au anomalies with on the stage set in particular I think it would be interesting too if there was some geospatial data or IP address data which is not stored on the chain right but you can see that from the mempool if you're operating a node and is there some like relationship between IP addresses and can you basically see an adversarial relationship between peers where they try to block one another and there's probably some interesting patterns in there if if that does happen so you're also not storing mempool transactions at any point I can't like query big this data sets okay what are the transactions waiting to be confirmed

it's only at the correct locks six blocks after height correct no mempool in bigquery data set that's correct okay cool so can you talk about the technical infrastructure that you've built in order to in order to query the blockchain and pull this data into your data set sure you want to talk about just Bitcoin you want to get into aetherium how do you want to look let's talk about the clones for the moment I mean we can talk about a theorem a bit later okay yeah sure so for the Bitcoin big point infrastructure what we did is we built a custom Bitcoin client with a library called Bitcoin J so this is the Java version that implements the Bitcoin peer-to-peer protocol and it's appear on

the on the network like any other peer and it accepts the new blocks coming in and if a peer asks for a block it will send the blocks out but we're not doing any mining we're just acting as sort of like a file sharing node like a BitTorrent node basically storing the blockchain and then we know when new blocks are coming in because we're accepting these new files right block files and we're looking at what the height is of the chain and every time a block comes in we increment this follower position that's X blocks behind and then kick off a job you something called cloud functions that will grab that block from cloud storage so there's the there's the node to

running a compute engine instance it's a virtual machine so it's like a computer no special mining hardware because we're not mining and then it writes the block file to storage and then there's a function that watches that storage area for a new file to have come in and it processes that file and sticks it into bigquery that's it that's all it does so you essentially have a node that's listening to the network that pulls in transactions and then stores them to into bigquery where they can then be created so that the bigquery as as a user how do I query it how do I create the blockchain what what language am i using are there API is we're sort of

SDKs that I can plug into my software it is query the Bitcoin blockchain yeah so we're using a language called SQL or sequel this is an industry standard language for interacting with databases so Oracle database runs sequel and you know my sequel Postgres Microsoft Access and all of that it's a it's an industry standard thing teradata all of them use they support some core sequel words or functions you could say like operators and then typically there's some vendor specific extensions so bigquery has some vendor specific extensions to related to AI geospatial various other things but we're working

with the the blockchain data you don't really need to have any vendor specific extensions you could conceivably take the the loading system and push it into my sequel and it would all work in the same way okay so you could construct a query as simple as you know select all transactions from this day to this day where the amount transacted was like one Bitcoin for example and it would just return all those all those transactions in a result okay and you could select the mean price of a transaction per day across all days or you could look at the quartiles or max or variance or whatever attribute you're looking to if we continue on with this per day example

you could partition by day you've got partition by block I mean there's there sort many different ways you could slice it I mean you could correlate it with some other public public data set like whether and try see if there's any like patterns that are affecting people's transaction volumes isn't that for sure so well we were prepared for this episode you you mentioned this platform called kaggle which I had sort of heard of before but you know wasn't super familiar with it so the way you described it to me is sort of this is github for data analysis so it's it's a platform where data scientists can share their own at their data analyses I presume sort of like queries and code

and you know they can fork them and it's sort of this open platform where this open community of data scientists talk about some of the things that people are doing on kaggle things that you may have noticed with regards to these data sets what type of analyses are people doing on the Bitcoin blockchain yeah so kaggle is it's the largest community of data scientists online so there's quite a lot of machine learning happening there they analyze in these notebook environments so there's a computer sitting behind a interactive forum in a web browser and they can run code against data that's sitting on a

remote machine that's connected to the web via the via the web browser and it can also connect to bigquery so they can pull data into this analysis environment and process it with code inside of this notebook environment they're typically programming in Python specifically in in regards to the the Bitcoin data set users have mostly been interested in in looking at these features that we were just talking about like prices of transactions denominated in Satoshi's or what were the largest transactions per day and then correlating these to other data sets so they like to link against other private data so for example as part of the Bitcoin data set as you mentioned we just have the blocks in the

transactions it's really just the chain data right but since all financial stuff frequently people want a link against financial data and so they'll bring in some other tables they may host it in their own bigquery tables or they may be uploading a CSV as part of the part of their analysis and then they can start doing you know pricing type analyses over time interesting so then what what what people are analyzing is a Bitcoin transaction data and in crossing that data with other could be financial data for example if one wanted to see if there's any correlation between like the Nasdaq financial index these are Dow Jones with the price of Bitcoin for example why

suppose another price of Bitcoin wouldn't because it's not it's not in your data set but you know one could make that that type of analysis using using the bigquery language perhaps combined flecked some machine learning stuff yep yeah Xillia tables that are available elsewhere the whole I think there's there's you know the more data that you can link together that's structured and documented and linkable the more value that comes out and it's not just one plus one equals two it's more like well it's not two plus two because how would I say three plus three equals six it's three times three equals nine right your utility of the data is more of a product of the pieces as

opposed to the sum if you've listened to previous episodes with Marley Grey and Matt Turner you know that Microsoft is committed to providing enterprise-grade tools and infrastructure for blockchain developers well the azure blockchain workbench is perfect for organizations building consortium networks take the etherium proof of authority template for example it's ideal for permission networks where consensus participants are known and reputable aetherium on Azure has on chain network governance that leverages parodies extensible proof Authority client each consortium member has the power to govern the network or delegate their consensus participants to a trusted operator and parodies webassembly support allows developers to write smart contracts in familiar languages like C C++ and rust

as your blockchain workbench was created on the same principles that drive all production services and Azure so you know you're relying on secure redundant infrastructure that can scale and with built-in services like authenticating api's off chain databases and secure key management services you can scaffold your infrastructure in just a few hours to learn more about Azure blockchain workbench and how Microsoft is advancing blockchain usability and enterprise check out AKMs slash epicenter and start building today we'd like to thank Microsoft Azure for their support of epicenter so recently there was a blog post that was published sometime in August that announced that Google was also releasing an etherium data set yeah that's now available on bigquery so what

has the reception been like it's been initially very positive this is only a few weeks ago now not even a month yet but the numbers are looking good in terms of utilization and number of inbound increase I get a lot of basically direct things from developers because my my name is out there I'm on Twitter etc and when they you know some fraction of them want to talk to me about things they're interested in doing and relative to the Bitcoin data set the amount of developer activity has been very high so I expect that the utilization of the etherium data public dataset will be will be even larger than the Bitcoin public data set yeah the Bitcoin one it's been regularly heavily scanned ever since it was released so

it's a very popular public data set and presumably people are acquiring this public data set to link against their private private data right to do I don't know what they're trying to do something analyze it for some purpose the etherium one because it's such a large developer community though I think there's gonna be many more different there's going to be more variety of applications and maybe even more more volume of applications on this one just because it's a lot more complex so what are the unique challenges that that you face in implementing the etherium data set as opposed to to Bitcoin I mean the Bitcoin data set that it seems like I mean I don't wanna say it's a simple feat but

it seems quite simple right you've got a node it transactions there's some profit like some data processing in these transactions to normalize the data and then it's put into a it's put into a essentially a sequel database with etherium there's a bit more to it there's the transactions but there's also six modes smart contract transactions and and token transactions and there's a whole bunch of other things that go into that that you know add some complexities you can you you talk about those on how you've overcome those challenges yeah so as you said Bitcoin data set is really just you know transfers and then the cost of the transfer that the requester was willing to pay for that transaction right and there's some strange like obfuscation

where there's change addresses and intentional obfuscation of who's really paying whom as part of like a pseudo private design etherium doesn't have that so this analysis of where money is flowing to is not a problem in etherium but there's this other difficulty where where there can be data that go along with the transaction and that could be a smart contract or other types of data and this that's that's really the core complexity is you've got this aetherium virtual machine that takes inputs that go into some compiled code that lives at an address on the on the blockchain that can do things with that input those

input bytes that are going along with the transaction they're coming in and what those smart contracts do with those input bytes is arbitrarily complex it's the it's a Turing machine so dealing with representing that complexity is very difficult which is why it was it took a lot longer to to get the etherium data set release than the Bitcoin did is say so token transfers that's a great example the the infrastructure for the etherium data set is pretty similar so we're operating an aetherium node and it's writing out some files into cloud storage but it it deviates from the Bitcoin design there where the the loading is happening not just as the

direct insertion into bigquery but there's another another cloud component call cloud composer so this is based on an open-source project called Apache airflow that you can define an ETL pipeline so ETL is a term used in data warehousing basically everything we're talking about today is data warehousing and then analysis on the stored data so ETL is for extract transform and load and basically you're extracting data from the etherium node you're transforming it to some form that will be useful for users so it could be you know reading the transactions and parsing them so that you can see if it's the ERC 20 transfer or not or ERC 721 transfer or whatever other kind of smart

contract function call and then the load part is putting it into the into the tables and so there's a whole bunch of additional ETL processing that we're doing as part of loading the etherium data into bigquery because we don't just want to load the transactions where we give the only the bytes that were the input and then they go to some smart contract but we don't tell you what did that mean there's no at at face value there's really no analysis that's interesting analyses that are enabled by only giving input bytes you actually have to look at what the smart contracts are doing and so we're looking at these things called traces and logs that are emitted by the smart contracts as part

of their operation so they they have some events that are coming out they describe what the smart contract is doing and so we're putting all of that into some tables to so that you can you can look at those and aggregate on the the effects of the smart contract on the network can you go a bit more detail about how you perform these analyses on on these traces and logs yeah sure so this is getting pretty far down into the weeds of how aetherium works but smart contracts they have these functions that are defined in typically solidity and a given function will have some defined inputs an input could be like an address or it could be an amount or it could be

other random binary data and then they do something with this data and what they're doing is the result of that is being emitted as the event typically and so what we're doing is we're taking those events and putting them into a table so you can so you can observe it an example would be and the input is actually very difficult this is difficult to understand because it's it's a binary string it's a bunch of bytes and you need to segment the bytes according to the function specification so there's like all this array manipulation and very low level stuff that's not really relevant to the business purpose of making a query or

analysis that you have to do in order to be able to do the query or the analysis and so really what we're doing is we're we're factoring out all of this menial labor that a developer would have to do doing it once and doing it correctly and then nobody has to work on that problem again if they're willing to you know run their operations on bigquery so the theorem blockchain contains as the data is contained there so within the input bytes of the smart contract you know we we we have we have the information there for example perhaps some address data or the the the function call hash and what you're doing is you're extracting that data from

essentially just a blob of bytes and making it available in this query format so that now you might be able to say okay what are all the contracts that are calling this specific function narek or what are the transactions that are calling to this specific function in this contract and making it easily available whereas if you want to do that by yourself you would have to build it from scratch yeah we're just reformatting the data right it's in this it's in this form that is just difficult to difficult to access for doing certain types of analyses it's designed for usability by the etherium blockchain software right

cerium peer-to-peer network is concerned with consensus and concerned with the efficiency of transactions and concerned with the operations of the etherium virtual machine in the blockchain right but it's not concerned with hey what if somebody wants to do some historical analysis on all the data in here that's irrelevant from the point of view of a database if somebody was to design a database that is for transactions then you would design it in a particular way if you had a database you wanted for analysis you would design that in an almost like you would call it the opposite way and if you get into data warehousing theory there's these are the two extremes of different types of database design one is called an OLTP it's an online transaction processing

database the canonical example is usually like a hotel or an airline reservation system it's very concerned with transactional integrity and very concerned with a throughput number of transactions per unit time but it doesn't really concern itself with analyzing like price trends of the hotel rooms or the flights right but you can take that transactional data and reformat it into an online analytics processing system which is an OLAP database which D normalizes it doesn't care about transactions but tries to structure the data in such a way that it's easy to send any arbitrary query against it and get reasonable

performance for the query when you want to get the data back to a business application or an analyst or something like that so at a higher level that's what we're doing with this blockchain data we're taking loyalty to data it's optimized by the OLTP system ie the blockchain system for transaction throughput and we reformat it and we make it available as OLAP okay so it occurs to me that an OLTP system is like Bitcoin but without mutability Bitcoin does not have mutability yeah these LTPS can be yeah so if you remove if you add an immutability constraint to an OLTP you get a blockchain sure you could say it like that okay what type of analyses have you been

doing on this on a theorem data says in the blog post was a couple of examples there can you talk about those yeah one of the examples I mean some of them are time series examples where we're looking at you know number of transactions per day and that's a very obvious kind of thing to do what interests me more though is the characteristics of actors on the network or I guess in turr actors you could say because wallets aren't doing anything on their own on Bitcoin or at the areum or any of these blog any other block chains there's always some interacting partner right and that interaction between those two partners tells you something about it's some

measurable observation and if you look at multiple these observations in aggregate for one address or groups of addresses over time you can start to quantify what they're doing and assign attributes to the to the addresses so you could begin to identify as a concrete example you could identify exchanges for example because they're typically going to have large volumes flowing in and out of them and they'll typically have many many interacting partners of other wallets sending sending money or tokens in or out now there could be other addresses in the network that aren't exchanges or aren't

known exchanges that has similar behavior but we could use the duck typing approach to to characterize those I mean if it looks like an if it looks like a duck and it quacks like a duck and it smells like a duck it's probably a duck so you could start to label exchanges that are unknown based solely on their attributes by looking at their behavior over time mining pools would be under example you can start to pick those out and you could imagine you know what kind of characteristics the pool would have or a miner who is time-sharing inside of a pool they'll have a particular characteristic right they're going to receive deposits periodically and only after the mining pool mines a block right so that sort of

thing is very interesting to me because it relates very very much to the type of work I was doing in my dissertation as a graduate student I was looking at genetic networks and particularly the human human human genome Network which is composed of genes that are interacting with one another to operate a cell which is basically a distributed parallel highly parallel distributed system of these you know molecules that are interacting with another to process information and there's a whole bunch of analytical techniques that can be used that come from biostatistics for analyzing these biological networks you can analyze financial networks with the same techniques and it turns out some of these guys doing anti-money laundering

applications or other types of like fraud analytics or you know forensic accounting they're also very interested in this kind of stuff but they don't necessarily have the the level of sophistication the biologists do because the National Science Foundation and the National Institutes of Health has been throwing a lot of dollars at you know curing cancer for a long time which is where a lot of the why a lot of these methods were developed because cancer is a disease of the genetic program I feel like I'm kind of like getting off on some tangents here and rambling but there's some very direct connection like between the math and the methods of what I was doing as a graduate student and this stuff that's happening with

blockchain right now and not the block so that consensus themselves but the interaction between the entities on the network that's what I'm interested well let me feel free to elaborate I think it's really fascinating topic so yeah well I just gave you two examples one of them being identifying exchanges the other when being identifying mining pools or timeshare miners I don't know what other sorts of interesting patterns would you want to look for well well there was one example here in the block-post that that was was kind of interesting and that's analyzing the functionality of smart contracts can you dive into that one yeah sure that's that's pretty cool because so so as you mentioned earlier there's these hashes of the functions

right and each function has its own signature if you are to consider a function as having some set of a smart contract is having some set of functions available to it where it's between zero functions to whatever all possible functions in the hash space for billion functions and every smart contract has some subset of that 0 to 4 billion you can define a distance metric or a similarity metric between any two contracts that tells you how similar they are in the functions that they implement so it's reasonable to say that if if two smart contracts have the same functions available probably what they

do is if not the same quite similar so all our C 20s implement four methods and so you could find all year C 20 contracts by checking to see if the contract implements those four methods that's actually how we do this in the in the database there's a table that documents specifically ERC 20 transfers and there's a table that lists all smart contracts and there's a boolean column on that smart contract table which says is ERC 20 or is the rc7 21 because these are the two you know dominant smart contract types and so we we just kind of add as a convenience the analysis pre analyzed to save developers some time so they can only restrict analysis to those

smart contracts but you could do it for any arbitrary you know set of functions you're interested in all the data are there right so for any standardized smart contract such as the token contract or the the 721 a or C 721 contract you can essentially just look at the transaction look at the function hashes in the contract and determine whether or not that's a fact you know this type of transaction or not that's right and then you're putting that bit you're doing you're making that available so that you you don't have to you don't have to extract it yourself it's already made available for you in the bigquery news

yeah and I define a function for the similarity metric we're just describing there's an analysis like this that's done as one of the examples in the blog post where I show the original crypto Kitty's contract and you can see all of the subsequent iterations where they they basically upgrade the contract and they're all similar to one another because they're adding functions over time and then you can also see clones because the crypto Kitty's contract is open source you can see somebody made crypto puppies and crypto clowns and other you know variants of this thing that's basically a clone of the game using the same code and it shows up it's a very similar contract so if you have some you know game that you like to play it like let's say you like to play any of these you know connect three jewel

games candy crush or something you could find all other candy crush liked games on the blockchain because they would have similar functionality I think I saw this in one of your talks that you gave in Singapore recently I try to find a Lincoln and add to the show notes but there was this analysis of the frequency that smart contracts were specific contracts recalled in the token contract I mean I guess overshadowed everything else but then there was one one contract which more recently had gained quite a bit of volume and that was the crypto kiddies contract yeah it had a very brief spike and I think probably most of your listeners will remember that the the etherium blockchain like you

couldn't add transactions to it it was clogged up and a bunch of I cos had to delay because if there was a crypto Kitty craze right but er CDRC 20 transfer function is the most common one or has been I haven't to the data recently but I would imagine still it's still quite force' I'm sure I still is what one area that I wanted to dive into is so we mentioned earlier that the Google cloud services are integrated so you know we can we can essentially connect different services so what are the types of analyses that one could make I figure there's actually quite powerful analyses that you could

make using the the AI component of Google cloud with some with the a thorium transaction data have you thought about you know the types of things that one could infer from from this data by plugging in a machine learning algorithm and doing some deep learning on this transaction data yeah so I am actively thinking about that and so the data that are in bigquery there is a couple of first-order things that you can do just with the data as they are today you could look at the input bytes going into into transactions and begin to reason about what the functionality of a contract might be or

an functions function signature that you don't know what it does but you see what its inputs are you could identify round numbers or you could identify addresses and this would give you some hint as to what that function is likely to be to be doing without getting into analysis of the stack trace of the virtual machine you can also analyze the smart contracts themselves right because these are all some byte code and you can treat each of those bytes as features and train some kind of analyzer to classify contracts it would probably be something quite similar to what we were just talking about a finding similar similar contracts but would have some additional ability to detect other things that are

difficult to do with such a simple method but the more interesting methods you can't really do them so let's come back to you know I was talking about networks and network analysis you can't really do network analysis on the bigquery data set directly because network analyses require traversals through the network so this basically would mean in scanning like looking at a table that is set up for scanning we talked about bigquery being a scanning system earlier it you would have to basically have random access and do these recursive queries which it's not really very well suited to that it's well-suited for linear

scans and so in order to do these analyses that involve traversing the network you need to move them into another type of database called a graph database and so what's beginning to happen now this is me and some other data scientists you know in the open source space we're exploring this and there's a another link we can put in the supplement to some work by one of my collaborators we're loading data into graph databases analyzing it reducing the graph down to something like a single measurement per address or per transaction allowing us to a assign a value between zero and one of the probability of this thing being in exchange for example and then taking those attribute data from the graph

debase putting them back into bigquery so that it becomes we would call it a vector of features this is getting kind of very specific into machine learning now but these these machine learning models typically want to operate on something called a vector space model so they walk to every observation to be a row but what I just told you is like in order to work with the network data it's not row like in nature its graph like in nature but you can reduce the graph down to rows by going out to a graph database so iterating between the data warehouse and a graph database to create more elements in the in the data warehouse basically doing enrichment through

analysis of the graph enables the these AI algorithms to begin analyzing the graph so that's the the direction that we are we are moving right now and fortunately in a Google cloud has good technologies for building graph databases as well so we're secure there but it requires a lot of work you mentioned earlier that that the reddit the entire entirety of reddit was also in the as a public dataset in in Google Cloud yeah it occurs to me that you know we might be able to do some sort of analysis as well as to the success of an ICO for example so take all the icy OS you know since since you know what - two or three years ago look

at the amount of money raised and then may perhaps correlate that with some natural and language processing data that you would have extracted from like reddit communities and maybe some other data in there and come up with some kind of predictive model to determine you know what types of characteristics what kind of like characteristics of communities that you might look for in order to determine whether an ICO will be like you know successful or not or this time type of thing yeah totally so reddit actually was presenting at the Google cloud next conference over over the summer talking about what they're doing to analyze some of the activity to build they're trying to basically help

people find more content on reddit so they can continue you know living out there happy reddit life but yeah you can use this type of technology called natural language processing it's a type of AI that tries to understand natural language like the Google assistant you might see this on your phone or Siri or Google duplex or Google home you've probably seen these like conversational agents right BOTS so you can you can take some text or speech to text and then take the text and send it into an AI and extract things like hey what were the key what was the key object that was being talked about here or what was the sentiment being expressed about that

object and quantifying it basically reducing the human readable text down to some numbers and then you could correlate you could then cross-reference those numbers so back to your example like okay was it more important to the success of a project that there was a lot of positive sentiment or is it more important that there was just a lot of buzz in general even though a whole bunch of it was negative maybe there's I mean I don't I don't know what the answer is but you could begin to explore that kind of line of reasoning by linking the yeah the reddit dataset to the to the Bitcoin or the coin or or aetherium dataset could make for a real interesting PhD thesis thesis I would you know love to support any data scientist who wants to

work on this please if you know anybody listening who wants to work on that shoot me an email or join us join up on Kaggle or you know let's get the data analysis going I can see something like analyzing memes you know analyzing also like there's that's like dogecoin memes you know like how many people how many times are people sharing a specific meme does that have an impact on the CEO or something in that nature I mean we even have this vision API which is it's the sibling of the language API that can actually look at images and analyze those and tell you what's in the image and you know what's the general sentiment about that yeah for sure well so looking into the future are there any plans to to release

other other tokens blockchains or cryptocurrencies as on the open platform I'm looking at a bunch of other stuff right now yeah more more of these public blockchain data it's all just sitting there and it's public datasets and the you know these current ones of getting getting a bunch of good traction analysis yeah that would be interesting to do the I think the etherium dataset has has quite a lot of mileage to be gotten out of it though it's just it's just so deep and interesting and there are yeah I've not done any plot of developer activity versus I don't know market cap or anything like that but I would imagine a theory would be a real

outlier that community has done a good job of making ecosystem components available and being very supportive of their developers and so there's just it's just very vibrant compared to a lot of the other projects even though Bitcoin is you know bigger by market cap it doesn't it didn't it didn't really attract as much I didn't I didn't get as many inbound enquiries about that data set as I did for free to get 3m data set well Alan this is all very fascinating I'm gonna thank you for coming on the show and talking about this I think that I think there's a lot of interesting things to be built on top of these data sets and you know we kind of mentioned this before the show but in in some ways this data is meant to be public right

and a lot a lot of companies out there is sort of made their business model upon these data sets I mean like block explorers companies to do you know blockchain analysis like their entire business models are built on what is essentially a public data set but where the blockchain itself doesn't really have the underlying infrastructure that allows just anyone to make these really new complex simple queries on top of them we've had to build all this other infrastructure on top so the fact that Google is making this available to the public is really great and I'm looking forward to seeing you know what kind of applications that we're kind of research comes out of this in the future yeah so we're gonna do we're here to help developers make cool applications great

so we will add I've got a ton of links tad to the show nuts so everything we've talked about the blog posts the links to the actual data sets and some of the other articles that we mentioned during the show we'll be in the show notes so thank you for our listeners for tuning in you can get new episodes representer every week by subscribing to us on YouTube SoundCloud you know your favorite podcast app whether that be on iTunes or Android we will soon be on Spotify that is gonna be finalized in the next couple of weeks I know that people have been asking for that and it took a while but we're finally gonna be on Spotify so we'll be we'll be tweeting that out as soon as as soon as that becomes the case I'll be like I'm that's great great awesome

we're also on Google Play for the student who don't know that we're also on Google Play Music and if you want to support the show well we love to hear from you by getting iTunes reviews it helps people find show and we're always happy to see your reviews there so thanks so much we look forward to being back [Music] [Music]