Artificial intelligence researchers in Israel looking for treatments for COVID-19 needed to study the records of thousands of early patients of the pandemic. Normally, the process of getting permission from these patients to access their confidential data for this research would have taken weeks or months, but the researchers were able to access the data almost instantly. The reason? The data they received was synthetic data: instead of the raw medical records of the patients, an Israeli company called MDClone recombined the original records into a new, statistically valid data set that the researchers could use without fear of breaching patient confidentiality.
Artificial intelligence systems that employ machine learning develop rules and inferences about the world that then guide decisions about new information. Machine learning depends on access to a sufficient amount of data about the application area to train the system and allow it to build a robust set of rules and inferences. The more data the system has from examples of a particular decision or situation, the better the model the system can build to provide intelligent and useful insights. However, there can be problems in acquiring the data the system needs.
Enter synthetic data. Synthetic data refers to data sets that contain records that mimic real-world data but are not actual real-world records. Any organization seeking to apply artificial intelligence, machine learning, and deep learning to its operations needs to be aware of the importance of synthetic data.
What Is Synthetic Data?
There are two sources for synthetic data:
- Real-world data. Real-world data can be stripped of personally identifiable information (PII) and personal health information (PHI), but that’s not sufficient to fully safeguard privacy because the data records can still be compared to other identifiable sources. As in the COVID-19 example, the anonymized data must be recombined in a way that preserves all of the statistical properties of the data set so that the machine learning algorithms can draw valid inferences and create valid rules.
- Simulated data. In some instances, the obstacle for machine learning is an insufficient supply of real-world data. Sometimes collecting real-world data would cost too much or take too long to be practical. In these cases, simulations can supply data that is sufficiently close to real-world examples that the machine learning algorithms can learn properly. For instance, the self-driving vehicle industry uses a combination of real-world sensor data from vehicles running on roadways and simulated data from driving simulations (even video games like Grand Theft Auto).
There are many reasons to use synthetic data instead of raw real-world data:
- Privacy, confidentiality, and other data usage restrictions, like HIPAA health privacy regulations in the US or GDPR consumer privacy protection in the European Union.
- Insufficient real-world data due to the cost or difficulty of collecting the data.
- Unencountered conditions, such as phenomena that have never been observed (like a supervolcano), places that have never been reached (for example, the surface of another planet), or just the operating conditions of a system that hasn’t been used yet.
- Correction for statistical anomalies or biases in the real-world data, as when there are rare outliers in the real-world data that need to be made more common artificially so the system has enough examples to train on.
Where Is Synthetic Data Used?
Synthetic data supports many different applications. Some of these are:
- Automated software testing for DevOps. Software development has always required test data, but today the short Agile development cycles of DevOps require more test data than ever.
- Self-driving vehicle development. Operating sensor cars on real roads is a costly and slow process, and synthesizing data from driving simulations provides a much bigger dataset for training self-driving AI.
- Manufacturing automation and robotics. Like automotive data collection, collection of real-world data in robotics and manufacturing applications can be slow and costly, so synthetic data can make training AI systems in these applications more efficient.
- Financial services. Like healthcare data, personal financial data is subject to tight confidentiality controls, and synthetic data gives developers and corporate users access to bigger datasets without violating privacy.
- Marketing simulations involving consumer behavior. Actual online behavior of consumers is subject to GDPR and other restrictions, so a synthetic dataset enables broader and deeper training of marketing AI.
- Clinical health research. PHI is highly regulated, so synthetic data makes AI and machine learning possible where datasets might otherwise be too restrictive to be useful.
- Facial recognition. Using photos of real people to train facial recognition can violate privacy restrictions and can lead to biases from underrepresented types of faces, and synthetic facial data can solve these problems.
- Social media. Social media platforms need to train AI systems to detect hate speech and extremist content, so they need datasets that aren’t subject to privacy regulations and concerns.
Enhance your skill set and give a boost to your career with the AI and ML Course.
Synthetic Data Helps AI Grow
Synthetic data is a rising area of research and development in the field of AI and machine learning. The Massachusetts Institute of Technology recently introduced its Synthetic Data Vault open-source project, an effort to provide a one-stop source of synthetic data for all kinds of machine learning applications. While the Synthetic Data Vault is new, it builds on research that has been ongoing at MIT since 2013.
The synthetic data field is growing in terms of a number of players as well. Here are ten companies in the business:
- AiFi for detail
- AI.Reverie for machine vision
- Anyverse to self-driving vehicles
- Cvedia for machine vision
- DataGen for augmented reality in interior environments
- Diveplane for clinical healthcare data
- Gretel creates a data synthesis tool
- Hazy for financial fraud detection
- Mostly AI for the banking, financial services, and insurance industry
- OneView for geospatial imaging
Synthetic data is not only creating opportunities at companies in that particular area, but for all applications of artificial intelligence, machine learning, and deep learning. The demand for AI architects, machine learning engineers, DevOps experts, and related technology professionals is growing quickly. Simplilearn’s courses and programs, like our AI and ML Course in partnership with Purdue University, will give you access to the skills you need to compete in this important field.
