Uploaded image for project: 'CloverETL'
  1. CloverETL
  2. CLO-2588

Dedup support for unsorted input

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: rel-3-3-0, rel-3-5-0-M2
    • Fix Version/s: rel-4-3-0-M1
    • Component/s: Engine
    • Security Level: Users (General product issues)
    • QA Testing:
      Graph automated test
    • QA Test Identification:
      after-commit.ts/Dedup_unsorted*
    • Sprint:
      PRG Sprint 15, PRG Sprint 16, PRG Sprint 17, PRG Sprint 18

      Description

      Please implement dedup strategy for unsorted input using HashMap (similarly as we have it done for Aggregate). New attribute "sorted input" with default value = true should be added to control dedup strategy.

      Justification: this is extremely efficient for large datasets with a low number of unique values; it is much more efficient than sorting.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                korbelj Jan Korbel
                Reporter:
                ulrychj Jan Ulrych (Inactive)
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 days
                  3d