--- title: "Duplicate Detection" canonical: "https://support.soutron.net/space/HG/3277899/Duplicate%20Detection" format: markdown --- Duplicate Detection can be enabled in two modes: 1. Manual record creation 2. Record Import from a file Duplicate detection is not enforced in the Auto cataloguing module, but it is possible to enable record updating if matches are found for imported auto cat records. --- ## Manual Record Detection ### Setup To use manual record detection, open **System Configuration Maintenance** and enable the option '**Enable Manually created record Duplication Check'**. You can also set** 'Duplicate detection match (%)' **to your preferred value; the default setting is 90%. Once this is done, open **'Field Maintenance'** and set the **'Duplicate Detection Weight'** for each field you wish to include in the duplicate detection matching process. Values should be in the decimal range of 0-1. If you want to only match with a full value, simply enter 1. If you wanted to match titles where only some of the words match, enter a value in the range of 0.5-0.8. ### Manual Record Detection when cataloguing Once this is complete, you will see a pop-up prompt every time you create a new record that matches an existing title based on your settings and select the save new record button. This will provide you with a display of the record(s) that the system has found as a potential match. You can then decide to continue to save the record or cancel the new record creation. ![image](media://38fd38f4-a449-4f73-a298-0255b746b7ea) ## Record Import Detection File imports continue to be configured via Import Maintenance, however there are some additional options in the set-up page. A new column has been added which now allows you to specify the duplicate detection value for each mapped field. Setting this value overrides field maintenance. Values here should be entered in the percentage range of 0-100, with 100 being a complete match. This column is indicated by the label **'DD'**. At the bottom of the import template, there are two additional options under the 'Import Options' section. **'Run duplication detection when import completes' **- When enabled, this option will perform the duplicate check as part of the import process.  **'Duplication detect level (1-100%)'** - This value overrides the Duplication detection match value you set in the system configuration. This allows you to define different match rates for manual and imported records.   Once the template is configured, you can elect to run the import, either via a scheduled task or by using the 'run now' option. ### Running via Run now The 'run now' option now has an additional tab for 'Duplicate Detection'. This only has one option listed, which, when enabled, will ensure duplicates are matched upon import. This option can be enabled or disabled, whether or not it was selected in the template, and so you can be selective as to which duplicate detection runs for certain imports. ### Managing duplicated records found during import Once the import is complete, you can navigate to **Modules > Cataloguing > Duplications**. This page will display the details from all records that the system believes are a match for an existing record. The top of the page will display the total number of duplicates waiting to be processed, below which you can see the import template the record was imported by and the date & time-stamp of the import. Below these details is the imported record display (of the potentially duplicated record). To the right of the record display, you will see the record that the system has identified as an existing match. If several records are thought to be matched, you will be able to navigate through them all. ![image](media://0dd4f6b5-df10-4e88-b42f-73b4a1c1861c) Once you have reviewed the two records, you have four options: 1. **Add duplicate** - This will save the duplicated record currently displayed on screen into the system and give it a new unique CID. 2. **Delete duplicate** - This will delete the duplicate currently displayed on screen. 3. **Approve all** - If you are confident that none of the records matched and displayed are duplicates, you can opt to save all records into the system; each record will be allocated a new unique CID number. 4. **Delete all** - If you are confident that all of the records matched and displayed are duplicates, you can opt to delete all of the records from duplicate maintenance. ## How Duplicates are calculated The actual formula used to calculate the duplicate weighting is quite complex. As an example: The weighting assigned to each field is converted to a whole number: if that field is then matched, it is added to a total weighting. If that total weight equals or is greater than the match % set in system configuration or the import template, then the record is treated as a match. For example: - Title is assigned a value of 70% - Edition is assigned a value of 15% - Publication Date is assigned a value of 20% - Match percentage is set to 90% When this record is created, if the Title and Edition match, but the Publication Date is different, the total for this record will be calculated as: 70+15, giving a match total of 85. Because the match percent is set to 90, this record will not be recognized as a match.