How to Detect and Remove Duplicate Records in SQL Server

Duplicate records in SQL Server can lead to inaccurate reporting, data inconsistencies, and performance issues. In this article, we’ll go over how to identify and safely remove duplicate rows while keeping at least one unique record.

Detecting Duplicates

To find duplicate records in a table, use the GROUP BY and HAVING clauses to count occurrences of each unique combination of values:

SELECT column1, column2, COUNT(*)
FROM YourTable
GROUP BY column1, column2
HAVING COUNT(*) > 1;

Replace column1, column2 with the columns that define a duplicate in your dataset.

If you need to see the actual duplicate rows, use a ROW_NUMBER() approach:

SELECT *
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY column1, column2 ORDER BY id) AS row_num
    FROM YourTable
) t
WHERE row_num > 1;

Here, id should be a unique column to order the duplicates.

Removing Duplicates

Method 1: Using ROW_NUMBER()

The safest way to remove duplicates while keeping one unique record is by using ROW_NUMBER().

WITH CTE AS (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY column1, column2 ORDER BY id) AS row_num
    FROM YourTable
)
DELETE FROM CTE WHERE row_num > 1;

This deletes all duplicate records while keeping the first occurrence.

Method 2: Using DISTINCT INTO a New Table

If you want to be extra cautious, create a new table with only unique records:

SELECT DISTINCT * INTO NewTable FROM YourTable;

Then, drop the old table and rename NewTable back to YourTable.

Final Thoughts

Always backup your data before running delete operations to prevent accidental data loss. By regularly cleaning up duplicates, you can keep your SQL Server database efficient and reliable.

1
26

Related

Reading a file line by line is useful when handling large files without loading everything into memory at once.

✅ Best Practice: Use File.ReadLines() which is more memory efficient.

Example

foreach (string line in File.ReadLines("file.txt"))
{
    Console.WriteLine(line);
}

Why use ReadLines()?

Reads one line at a time, reducing overall memory usage. Ideal for large files (e.g., logs, CSVs).

Alternative: Use StreamReader (More Control)

For scenarios where you need custom processing while reading the contents of the file:

using (StreamReader reader = new StreamReader("file.txt"))
{
    string? line;
    while ((line = reader.ReadLine()) != null)
    {
        Console.WriteLine(line);
    }
}

Why use StreamReader?

Lets you handle exceptions, encoding, and buffering. Supports custom processing (e.g., search for a keyword while reading).

When to Use ReadAllLines()? If you need all lines at once, use:

string[] lines = File.ReadAllLines("file.txt");

Caution: Loads the entire file into memory—avoid for large files!

3
243

Primary constructors, introduced in C# 12, offer a more concise way to define class parameters and initialize fields.

This feature reduces boilerplate code and makes classes more readable.

Traditional Approach vs Primary Constructor

Before primary constructors, you would likely write something like the following:

public class UserService
{
    private readonly ILogger _logger;
    private readonly IUserRepository _repository;

    public UserService(ILogger logger, IUserRepository repository)
    {
        _logger = logger;
        _repository = repository;
    }

    public async Task<User> GetUserById(int id)
    {
        _logger.LogInformation("Fetching user {Id}", id);
        return await _repository.GetByIdAsync(id);
    }
}

With primary constructors, this becomes:

public class UserService(ILogger logger, IUserRepository repository)
{
    public async Task<User> GetUserById(int id)
    {
        logger.LogInformation("Fetching user {Id}", id);
        return await repository.GetByIdAsync(id);
    }
}

Key Benefits

  1. Reduced Boilerplate: No need to declare private fields and write constructor assignments
  2. Parameters Available Throughout: Constructor parameters are accessible in all instance methods
  3. Immutability by Default: Parameters are effectively readonly without explicit declaration

Real-World Example

Here's a practical example using primary constructors with dependency injection:

public class OrderProcessor(
    IOrderRepository orderRepo,
    IPaymentService paymentService,
    ILogger<OrderProcessor> logger)
{
    public async Task<OrderResult> ProcessOrder(Order order)
    {
        try
        {
            logger.LogInformation("Processing order {OrderId}", order.Id);
            
            var paymentResult = await paymentService.ProcessPayment(order.Payment);
            if (!paymentResult.Success)
            {
                return new OrderResult(false, "Payment failed");
            }

            await orderRepo.SaveOrder(order);
            return new OrderResult(true, "Order processed successfully");
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Failed to process order {OrderId}", order.Id);
            throw;
        }
    }
}

Tips and Best Practices

  1. Use primary constructors when the class primarily needs dependencies for its methods
  2. Combine with records for immutable data types:
public record Customer(string Name, string Email)
{
    public string FormattedEmail => $"{Name} <{Email}>";
}
  1. Consider traditional constructors for complex initialization logic

Primary constructors provide a cleaner, more maintainable way to write C# classes, especially when working with dependency injection and simple data objects.

0
68

Storing passwords as plain text is dangerous. Instead, you should hash them using a strong, slow hashing algorithm like BCrypt, which includes built-in salting and resistance to brute-force attacks.

Step 1: Install BCrypt NuGet Package

Before using BCrypt, install the BCrypt.Net-Next package:

dotnet add package BCrypt.Net-Next

or via NuGet Package Manager:

Install-Package BCrypt.Net-Next

Step 2: Hash a Password

Use BCrypt.HashPassword() to securely hash a password before storing it:

using BCrypt.Net;

string password = "mySecurePassword123";
string hashedPassword = BCrypt.HashPassword(password);

Console.WriteLine(hashedPassword); // Output: $2a$12$...

Step 3: Verify a Password

To check a user's login attempt, use BCrypt.Verify():

bool isMatch = BCrypt.Verify("mySecurePassword123", hashedPassword);
Console.WriteLine(isMatch); // Output: True

Ensuring proper hashing should be at the top of your list when it comes to building authentication systems.

2
226