Dublin Core The Dublin Core metadata element set is common to all Omeka records, including items, files, and collections. For more information see, http://dublincore.org/documents/dces/. Title A name given to the resource Conference Papers Conference Paper Faculty Publications- Conference Papers Dublin Core The Dublin Core metadata element set is common to all Omeka records, including items, files, and collections. For more information see, http://dublincore.org/documents/dces/. Title A name given to the resource File Validation intheData Ingestion Process Using Apache NiFi Subject The topic of the resource Apache NiFi; Custom processor; Data ingestion; File validation; Frequency validation Description An account of the resource In the industries of today, development and maintenance of data pipelines is of paramount importance. With large volumes of data being generated across industries on a continuous basis, there is a growing need to process and store this ingested data in a fast, and efficient manner. Apache NiFi is one such tool which possesses crucial capabilities that can be used to enhance, modify, and automate data pipelines. However, automation of the ingestion process creates certain inherent issues which, without being resolved, tend to be detrimental to the entire ingestion process. These issues vary in nature, ranging from corrupted data to changes in the file schema, to name a few. In this paper, a solution to this problem is proposed. By exploiting Apache NiFis custom processor development capabilities, problem-specific processors can be designed and deployed which can ensure accurate validation of the ingestion process on a real-time basis. To demonstrate this, two processors were developed as a proof-of-concept, which tackle specific file-related validation issues in the ingestion processthat of the file size, and, the ingestion frequency. These custom-built processors are designed to be inserted into the pipeline at key points to ensure that the ingested data is validated against certain standards and requirements. Having successfully demonstrated its capabilities, the paper presents the exploitation of Apache NiFis custom processor capabilities as a potential way forward to resolve the plethora of ingestion issues in industry, today. The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024. Creator An entity primarily responsible for making the resource Irfan M.; Gangadhar A.; George J. Source A related resource from which the described resource is derived Lecture Notes in Networks and Systems, Vol-922 LNNS, pp. 299-310. Publisher An entity responsible for making the resource available Springer Science and Business Media Deutschland GmbH Date A point or period of time associated with an event in the lifecycle of the resource 2024-01-01 Identifier An unambiguous reference to the resource within a given context <a href="https://doi.org/10.1007/978-981-97-0975-5_27" target="_blank" rel="noreferrer noopener">https://doi.org/10.1007/978-981-97-0975-5_27</a> <br /><br /><a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85197244973&amp;doi=10.1007%2F978-981-97-0975-5_27&amp;partnerID=40&amp;md5=03979cda62fcb4fad86937037c4ddff7" target="_blank" rel="noreferrer noopener">https://www.scopus.com/inward/record.uri?eid=2-s2.0-85197244973&amp;doi=10.1007%2f978-981-97-0975-5_27&amp;partnerID=40&amp;md5=03979cda62fcb4fad86937037c4ddff7</a> Rights Information about rights held in and over the resource Restricted Access Relation A related resource ISSN: 23673370; ISBN: 978-981970974-8 Format The file format, physical medium, or dimensions of the resource Online Language A language of the resource English Type The nature or genre of the resource Conference paper Coverage The spatial or temporal topic of the resource, the spatial applicability of the resource, or the jurisdiction under which the resource is relevant Irfan M., CHRIST (Deemed to be University), Bangalore, 560029, India; Gangadhar A., Binghamton University, State University of New York, Binghamton, 13902, NY, United States; George J., CHRIST (Deemed to be University), Bangalore, 560029, India