Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity Resolution
Wenpeng Yin, Shelby Heinecke, Jia Li
Nitish Shirish Keskar, Michael Jones, Shouzhong Shi
Stanislav Georgiev, Kurt Milich, Joseph Esposito, Caiming Xiong
The distribution gap between training datasets and data encountered in production is well acknowledged. Training datasets are often constructed over a fixed period of time and by carefully curating the data to be labeled. Thus, training datasets may not contain all possible variations of data that could be encountered in real-world production environments. Tasked with building an entity resolution system – a model that identifies and consolidates data points that represent the same person – our first model exhibited a clear training-production performance gap. In this case study, we discuss our human-in-the-loop enabled, data-centric solution to closing the training-production performance divergence. We conclude with takeaways that apply to data-centric learning at large.