SONARJAVA-6768 Implement new rule S7438: Incompatible bit masks should not be used in comparisons

[romainbrenguier]

Detect incompatible bit masks in equality comparisons where bitwise AND or OR operations are compared against values that are impossible given the mask, making the comparison always true or always false.

[hashicorp-vault-sonar-prod]

SONARJAVA-6768

[nathsou]

Nice rule idea! Found 3 issues in IncompatibleBitMaskCheck

[nathsou]

[P1] Do not truncate a long comparison operand

An int bitwise result is promoted to long for comparison; Java does not truncate the RHS. For if ((x & 0xFFFFFFFF) == 0xFFFFFFFFL), the condition is always false, but both values become -1 here and no issue is raised.

isIntOperation() returns true from the known-type branch whenever bitwiseOp.symbolType() is int, without checking whether the comparison operand is itself a long (unlike the heuristic fallback branch, which does call hasLongLiteral(comparisonValue)). Please preserve the comparison operand's width / account for the sign-extended int result in the known-type branches too.

[nathsou]

[P2] Unwrap parentheses before parsing literals

LiteralUtils.longLiteralValue only unwraps UNARY_MINUS/UNARY_PLUS before requiring INT_LITERAL/LONG_LITERAL; it never calls ExpressionUtils.skipParentheses. So if ((x & (1)) == 2) and if ((x & 1) == (2)) are silently skipped despite being always false. isIntLiteral/hasLongLiteral elsewhere in this file already call skipParentheses, so this looks like an oversight rather than intentional.

[nathsou]

[P2] Support unsigned high-bit long masks

Long.decode throws NumberFormatException for hex literals above Long.MAX_VALUE, such as 0xFFFF_FFFF_FFFF_FFFEL (this is even called out in a comment in LiteralUtils.java). Consequently if ((flags & 0xFFFF_FFFF_FFFF_FFFEL) == -1L) is missed, since longLiteralValue returns null and the check bails out via the mask == null || value == null guard.

[sonarqube-next]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0 Dependency risks
91.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[gitar-bot]

Code Review ✅ Approved 4 resolved / 4 findings

Implements new rule S7438 to detect incompatible bit masks in comparisons, addressing sign mismatches on int masks, missing sign extension in long comparisons, and truncation issues in no-semantic mode.

✅ 4 resolved

✅ Edge Case: Sign mismatch on int masks can cause false positives

📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:66-80
LiteralUtils.longLiteralValue decodes an int hex literal such as 0xFFFFFFFF to the positive long 4294967295, but at runtime int x & 0xFFFFFFFF treats the literal as the int value -1. Because isIncompatible compares the mask and value as longs without considering the int operand width, a comparison like (x & 0xFFFFFFFF) == -1 (which is true when x == -1) is reported as "always false". This is a false positive on a Blocker BUG rule. Consider restricting extracted mask/value handling to the operand's actual type (mask the value to 32 bits when the bitwise expression is int), or bail out when the constants require sign-extension reasoning; add test cases with negative literals and high-bit hex int masks.

✅ Edge Case: int hex mask not sign-extended in long comparisons → false positive

📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:59-62 📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:72-86
isIntOperation returns false for long-typed bitwise operations, so no width normalization is applied. But an int hex literal such as 0xFFFFFFFF used in a long operation is sign-extended by Java: (longX & 0xFFFFFFFF) really masks with -1L, not 4294967295L. extractMask/longLiteralValue decode 0xFFFFFFFF as 4294967295L, so if ((longX & 0xFFFFFFFF) == -1L) {} is incorrectly reported as "always false" even though it is true when longX == -1. Fix by sign-extending mask (and value) operands that are INT_LITERALs to long width regardless of the operation width, e.g. compute the mask as (long)(int)decoded when the mask operand is an int literal.

✅ Edge Case: Long bitwise ops truncated to int in no-semantic mode

📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:79-81 📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:58-61
The simplified isIntOperation returns !symbolType().is("long"). When semantics are unavailable, symbolType() is unknown and .is("long") returns false, so the operation is treated as int and both mask and value are truncated via intValue(). The old code guarded this case with hasLongLiteral, so long bitwise expressions using literals above the int range (e.g. (x | 0xFFFFFFFFFFL) == 0x10000000000L) are now silently truncated, producing incorrect results (missed detections) without semantics. Consider restoring the long-literal check for the unknown-type branch, e.g. treat as long when either bitwise operand or the comparison value is a LONG_LITERAL.

✅ Quality: signExtendedLongValue and maskOperandValue are identical

📄 java-checks/src/main/java/org/sonar/java/checks/IncompatibleBitMaskCheck.java:71-85
The two private static helpers signExtendedLongValue (L71-78) and maskOperandValue (L89-96) have byte-for-byte identical bodies (fetch longLiteralValue, sign-extend when isIntLiteral). Collapse them into a single helper to avoid divergence during future maintenance.

Implementation Status ✅ 1 / 1 issues implemented

✅ SONARJAVA-6768 — 1 / 1 objectives

The PR implements the new rule S7438 checking for incompatible bit masks in comparisons.

✅ 1 complete

• ✅ Implement new rule S7438: Incompatible bit masks should not be used in comparisons

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}