I am a senior researcher at the European Center for Algorithmic Transparency (ECAT) where I bring scientific and technical expertise to the European Commission in support of policy in various contexts including Internet intermediaries and online platform regulation ( Digital Services Act - DSA) as well as artificial intelligence technology regulation (AI-Act).
In short, my research (both at ECAT and prior to joining it) examines problems in the intersection of technology and society. Technology harbors exciting opportunities but at the same time, our ever-increasing reliance there on introduces complex societal problems, some of which can even turn into systemic issues. People have lost life savings due to crypto scams or online cybercrime, insecure and easily hackable IoT devices like routers and home storage devices have caused havoc and Internet outages, election outcomes affected through micro-targeted online political advertising, including via disinformation and deepfakes, not to mention the recent AI advances that are leading to phenomena like deepfake/revenge porn, and ultra realistic ransom calls in the voices of loved ones just to name a few instances. These are all examples of the types of complex problems that are created or exacerbated through tech, are extremely difficult to tackle, and require interdisciplinary and evidence-based remedies and rectification. My research typically approaches such problems by blending computer science - in particular cybersecurity and Internet measurement techniques - with other research from broader interdisciplinary areas of inquiry such as computational social sciences and economics to help measure their spread, harms, identify their roots, and to find potential mitigations or solutions whether through technical means, or non-technical means such as aligning incentives, soft policy and/or hard regulation.
Prior to taking on my current position at ECAT, I held a post-doctoral research position at the university of Amsterdam where I conducted research under the umbrella of ICDS, a joint research initiative from the University of Amsterdam's Institute for Information Law (IVIR) and the university's School of Communication Research (ASCoR) that focuses on the way AI and algorithms affect society. Prior to that, I held a post-doctoral research position at Delft university of Technology's cybersecurity group where I focused on the (cybersecurity) problems that arise from our ubiquitous reliance on information and communication technologies in relation to the role of Internet intermediaries in tackling them, for instance the structural insecurity problems brought about by insecure Internet-of-Things (IoT) devices that are flooding consumer markets. My doctoral research similarly examines methods for measuring and quantifying the effectiveness of Internet intermediary security practices through the lens of data on harmful content such as malware, phishing websites and botnets.
My research has been published in top-tier academic venues and covered in both national (Dutch) and International news and media outlets (see News section below). Some of this work has also been applied by the Dutch high-tech crime police unit, in legal hearings, and continues to benefit policy making. For an overview of some of my peer-reviewed research please refer to the publication list below.
My Research
Coverage of my Research/Projects + Personal News
A recent study on shortcomings in AI benchmarking that I was invovled with seems to have struck a nerve. We are seeing more and more coverage and references to our meta-review study in which we ask whether AI benchmarks can be trusted and discuss some of their major shortcomings.
AI benchmarks increasingly play a fundamental role in evaluating the performance, capability, and safety of AI models and systems despite their many shortcomings. We highlight many of their shortcomings in our study including (1) systemic problems like misaligned incentives, construct validity issues, unknown unknowns, and problems with the gaming of benchmark results or (2) sociotechnical issues like an over-focus on evaluating text-based AI models and one-time testing logics, or even (3) fine-grained issues in the design and application of benchmarks like biases in dataset creation, inadequate documentation, data contamination, and failures to distinguish signal from noise.
A recent article by the the Register titled "Why AI Benchmarks Suck" for has focused on our recent meta-review study by putting the spotlight on these issues. Similarly, In their article titled "Figuring out which AI model is right for you is harder than you think", Business Insider also covers our work quoting us as "The researchers said there are 'systemic flaws in current benchmarking practices,' which are 'fundamentally shaped by cultural, commercial and competitive dynamics that often prioritize state-of-the-art performance at the expense of broader societal concerns.' Funnily enough there are even AI generated summaries of our article circulating on Youtube.
For those interested in diving deeper into our study, the research paper is openly available and titled "Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation" (link below). This research done for this study was part of work that I have been involved in at the European Center for Algorithmic Transparency (ECAT) in support of the European Commission's work on the AI Act together with my colleagues and coauthors. I have included several links that coverage our recent study including a link to the study itself below.
- Link: Our study
- Link: The Register
- Link: Business Insider
- Link: The AI Evaluation Substack
- Link: Tech Brew
- Link: Stanford HAI
- Link: RAND
- Link: Sparky The AI Creator Youtube Channel
The dashboard was built with the aim of bringing about transparency into the use of micro-targeted advertising on social media platforms in the 2021 Dutch parliamentary elections and is based on data liberated from the political ad archives of social media platforms like Facebook, Instagram and Google. A hugely exciting part of this experiment is that we analyse and discuss the limitations of current social platform transparency practices and the the limited advertising data which they release to the public with a specific focus on how this could/should be improved. The limitations of current social media platform transparency practices can be better understood in a real-world practical setting, and our experiment can provide extremely useful input to European efforts on regulating social media platforms and limiting their potential societal harms.
Below some are links to some of the articles/stories covering and discussing the Dutch election dashboard, as well as other dashboards we helped build.
- Link: RTL Nieuws (in Dutch)
- Link: Het Financieele Dagblad (in Dutch)
- Link: Folia (in Dutch)
- Link: UvA Press Release
- Link: on Twitter
- Link: FTM Dashboard
Our Dutch election dashboard is also cited in the European Commission's Impact Assesment Report on regulatory proposals to improve transparency in political advertising. Transparency around online advertising particularly around what happens within walled-off platforms is increasingly an important problem that needs to be tackled through regulation. The commission's Report is available here: Link to Report (See p.91 for our dashboard work)
On the basis of my work and research the Dutch Ministry of Economics Affairs and SIDN Fonds have funded a project that tackles the prevalence of harmful content on Dutch Internet and hosting infrastructure through empirical measurements and establishing benchmarks for security. A platform (in Dutch) has been set up through which data and benchmarks on Dutch hosting providers' security efforts towards combating harmful content are communicated, including with the service providers themselves. The initiative incentivizes industry players to take more effective action against harmful content. Happy to see the results of my research come to fruition.
The New Yorker as well as the Dutch NRC, have published long pieces discussing of the phenomenon of `Bullet-Proof Hosting' that shine a spot light on my research into this phenomenon.
Bullet-Proof hosters are part of the hosting market where its owners/operators knowingly allow abusive content to be hosted and served online, and even actively put up resistance against takedown efforts, hence the name. The "bullet-proof" term is derived from a parallel drawn with providing “body armor” to protect against attacks. Apparently operators' of bullet-proof hosting services view takedown efforts by law enforcement and other defenders as attacks. In reality, their services enable a wide range of cybercrime. For instance they allow criminals to host some of their most valuable resources, such as botnet command-and-control (C&C) assets, exploit-kits, phishing websites, drop sites, or even host child sexual abuse material.
Both articles feature an interviews with Michel van Eeten, my PhD advisor, in which he discusses research that I led in collaboration with the Dutch high-tech crime police into a notorious Bullet-Proof hosting company that was legally taken offline. The research article that I published on the basis of its work is a first of its kind, thanks to the collaboration with the Dutch high-tech crime police and access to the seized data from the bullet-proof hosting company.It provides many previously unknown insights into how bullet-proof hosting companies operate based on ground-truth data as well as a systematic analysis of the effective ways to counter the harms that they enable.
The New Yorker and NRC articles are well worth the read, because they also paint very colorful stories with much more exciting details that usually also don't end up being told in academic research articles, although very much part of the picture.
- Link: The Study
- Link: New Yorker Article
- Link: NRC Article (Paywalled)
- Link: Archived Version of NRC Article
Brian Krebs of the infamous Krebs on Security blog has covered my research article on bullet-proof hosting in a fascinating post discussing some of the real world challenges of dealing with the phonemonon of bullet-proof hosting. Criminals that willingly provide hosting services on the Internet and thereby facilitate lots of cybercrime. One of my main findings in this work is that the bullet-proof hosting business seems to be not so profitable as many have imagined them to be. My research is the first to have looked at the finances of such a business.
What I have also come to realize after doing this research is that the `Economics of Cybersecurity' needs to have a harder look at and understanding of how cybercriminals make money. Something which interestingly enough some key figures of the field like Prof Ross Anderson and colleagues have also been apparently recently thinking of and publishing about: "Cybercrime gangs as tech startups"
Code & Datasets
As a researcher I have developed or contributed to the development of tools which may be useful to others researchers and engineers. The most useful of these is probably pyasn. Pyasn is a python package that, allows one to historically lookup IP address to Autonomous System mappings from raw BGP data. In simpler language to find out in which network an IP addresses resided over different points of time. Below you will find a link to its source code and other useful information.
More recently, I have been working on padsxiv. It is a tool to collect extensive data from social media platforms' so-called (political) ad-libraries. with the purpose of making data collection, archiving, and analysis more accessible and easy. padsXiv can for instance be used to collect data on all political ads that ran in the US 2020 elections on Facebook's platforms retrospectively, or to collect ad data on the current ongoing elections as events are happening. A distintive feature of the tool is that the data it collects includes the images and videos that run along side ads which is part of the data not provided by the platforms through their standard APIs by default, nor via other transparency tools. If you are interested in this tool feel please contact me as it is still underdevelopment and not publicly available (yet).
Along side padsxiv, I'm also currently working on an political ads archiving project, to automatically collect all political-ads from social media platforms using padsxiv, on a daily basis, and further enrich the data with automatically transcribed video text, as well as extracted information from ad images, to enable more deep topic and context analysis. and extract text format content form archived ad images. If this is something that interests you don't hesititate to contact me as I am looking for funding to make this happen on a large scale.